LLM Pre-training and Scaling Laws Cheat Sheet

Updated 2026-05-18

Next Topic: LLM Reasoning and Test-Time Compute Scaling Cheat Sheet

🧠Study flashcards on this topic64 cards · spaced repetition→

Pre-training large language models involves carefully balancing compute budget, data quality, model architecture, and training hyperparameters according to empirical scaling laws — most notably the Chinchilla scaling law, which reveals that for compute-optimal training, model parameters and training tokens should scale equally. This cheat sheet covers the full pre-training lifecycle: from tokenization strategies and data curation through distributed training frameworks, optimizer configuration, stability techniques, and monitoring, equipping practitioners with the knowledge to train models efficiently at scale.

What This Cheat Sheet Covers

This topic spans 14 focused tables and 71 indexed concepts, 64 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Scaling Laws and Compute BudgetTable 2: Tokenization StrategiesTable 3: Training Data CurationTable 4: Distributed Training FrameworksTable 5: Pre-training ObjectivesTable 6: Optimizer ConfigurationTable 7: Learning Rate SchedulesTable 8: Mixed Precision TrainingTable 9: Training StabilityTable 10: Architecture ConfigurationTable 11: Data Quality and DeduplicationTable 12: Monitoring and CheckpointingTable 13: Regularization and Overfitting PreventionTable 14: Batch Size and Training Dynamics

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Scaling Laws and Compute Budget

Understanding how model performance scales with compute, parameters, and data is foundational to efficient pre-training — Chinchilla scaling laws provide the formula for compute-optimal training, while Kaplan scaling laws describe the power-law relationship between loss and resources, guiding practitioners in allocating limited compute budgets.

Law	Example	Description
Chinchilla Scaling Law	$N_{opt} \propto C^{0.5}$ $D_{opt} \propto C^{0.5}$	• Model size and tokens scale equally with compute budget • for every doubling of model size, double training tokens — recommends ~20 tokens per parameter for compute-optimal training
Chinchilla Ratio	`70B params × 20 = 1.4T tokens`	• 20 tokens per parameter is the compute-optimal ratio • training Chinchilla 70B on 1.4 trillion tokens achieved better performance than Gopher 280B trained on 300B tokens
Kaplan Scaling Law	$L(N) = (N_c / N)^\alpha$ $\alpha \approx 0.076$	• Loss scales as a power law of model size • doubling parameters reduces loss predictably — earlier work that overemphasized model size relative to data
FLOPs Calculation	`FLOPs ≈ 6 × N × D` `# N=params, D=tokens`	• Total compute is ~6 × parameters × tokens for transformer pre-training • accounts for forward pass (2×), backward pass (4×) — used to budget training runs
Compute-Optimal Point	`If C = 1.2e24 FLOPs:` `N ≈ 1e11, D ≈ 2e12`	• The "Chinchilla optimal" point where neither over-parameterization nor under-training wastes compute • maximizes model quality per FLOP

Table 2: Tokenization Strategies

Tokenization converts raw text into discrete units that models can process — choosing between Byte-Pair Encoding, SentencePiece Unigram, and tiktoken impacts vocabulary size, compression ratio, multilingual support, and ultimately model efficiency at both training and inference time.

Method	Example	Description
Byte-Pair Encoding (BPE)	`"training" → ["train", "ing"]` `vocab_size = 50000`	• Merges most frequent character pairs iteratively until target vocabulary size reached • used in GPT-2, GPT-3, RoBERTa — deterministic, greedy algorithm

The rest of this section is part of the full cheat sheet

Table 3: Training Data Curation

High-quality, diverse training data is the foundation of capable LLMs — careful curation involves deduplication to prevent memorization, filtering to remove low-quality text, and mixing ratios to balance domains like code, web text, and books, ultimately determining model capabilities.

Technique	Example	Description
Exact Deduplication	`sha256_hash = hashlib.sha256(` `text.encode()).hexdigest()`	• Removes identical documents using cryptographic hashes • essential to prevent memorization and reduce training set size — GPU-accelerated deduplication can process petabytes

The rest of this section is part of the full cheat sheet

Table 4: Distributed Training Frameworks

Training LLMs at scale requires distributing computation across hundreds or thousands of GPUs using specialized frameworks that partition model weights, gradients, and optimizer states while minimizing communication overhead through techniques like pipeline parallelism and ZeRO optimization.

Framework	Example	Description
FSDP (PyTorch)	`from torch.distributed.fsdp` `model = FSDP(model)`	• Fully Sharded Data Parallel partitions model parameters, gradients, and optimizer states across GPUs • PyTorch's native implementation — easier to use than DeepSpeed ZeRO-3

The rest of this section is part of the full cheat sheet

Table 1: Scaling Laws and Compute Budget

Law	Example	Description
Chinchilla Scaling Law	$N_{opt} \propto C^{0.5}$ $D_{opt} \propto C^{0.5}$	• Model size and tokens scale equally with compute budget • for every doubling of model size, double training tokens — recommends ~20 tokens per parameter for compute-optimal training
Chinchilla Ratio	`70B params × 20 = 1.4T tokens`	• 20 tokens per parameter is the compute-optimal ratio • training Chinchilla 70B on 1.4 trillion tokens achieved better performance than Gopher 280B trained on 300B tokens
Kaplan Scaling Law	$L(N) = (N_c / N)^\alpha$ $\alpha \approx 0.076$	• Loss scales as a power law of model size • doubling parameters reduces loss predictably — earlier work that overemphasized model size relative to data
FLOPs Calculation	`FLOPs ≈ 6 × N × D` `# N=params, D=tokens`	• Total compute is ~6 × parameters × tokens for transformer pre-training • accounts for forward pass (2×), backward pass (4×) — used to budget training runs
Compute-Optimal Point	`If C = 1.2e24 FLOPs:` `N ≈ 1e11, D ≈ 2e12`	• The "Chinchilla optimal" point where neither over-parameterization nor under-training wastes compute • maximizes model quality per FLOP

Table 2: Tokenization Strategies

Method	Example	Description
Byte-Pair Encoding (BPE)	`"training" → ["train", "ing"]` `vocab_size = 50000`	• Merges most frequent character pairs iteratively until target vocabulary size reached • used in GPT-2, GPT-3, RoBERTa — deterministic, greedy algorithm

Table 3: Training Data Curation

Technique	Example	Description
Exact Deduplication	`sha256_hash = hashlib.sha256(` `text.encode()).hexdigest()`	• Removes identical documents using cryptographic hashes • essential to prevent memorization and reduce training set size — GPU-accelerated deduplication can process petabytes

Table 4: Distributed Training Frameworks

Framework	Example	Description
FSDP (PyTorch)	`from torch.distributed.fsdp` `model = FSDP(model)`	• Fully Sharded Data Parallel partitions model parameters, gradients, and optimizer states across GPUs • PyTorch's native implementation — easier to use than DeepSpeed ZeRO-3