## Table 1: Core Transformer Architecture | **Component** | **Example** | **Description** | |---|---|---| | [**Multi-head attention**](https://d2l.ai/chapter_attention-mechanisms-and-transformers/multihead-attention.html) | `heads = 8`
`Q, K, V = Linear(x, d_k)` | • Splits attention into **multiple parallel heads** that learn different representation subspaces
• each head computes scaled dot-product attention independently then concatenates results for richer feature capture. | | [**Self-attention**](https://jalammar.github.io/illustrated-transformer/) | `scores = Q @ K.T / sqrt(d_k)` | • Computes **query-key-value** relationships where each token attends to all positions in sequence
• forms foundation of transformer parallelization by replacing recurrence with direct position interactions. | | [**Cross-attention**](https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html) | `encoder_output → decoder` | • Used in **encoder-decoder models** where queries from decoder attend to keys/values from encoder
• enables translation and seq2seq tasks by bridging input-output representations. | | [**Causal (masked) attention**](https://d2l.ai/chapter_attention-mechanisms-and-transformers/transformer.html#decoder) | `mask = torch.triu(ones) * -inf` | • Prevents tokens from attending to **future positions** by applying upper-triangular mask during training
• critical for autoregressive generation in decoder-only architectures like GPT. | | [**Positional encoding**](https://kazemnejad.com/blog/transformer_architecture_positional_encoding/) | `PE = sin(pos/10000^(2i/d))` | • Injects **order information** into token embeddings via sinusoidal functions
• transformer architecture is permutation-invariant without explicit position signals. | | [**Feed-forward network (FFN)**](https://d2l.ai/chapter_attention-mechanisms-and-transformers/transformer.html#positionwise-feed-forward-networks) | `FFN(x) = ReLU(xW1 + b1)W2` | • Two-layer MLP applied **position-wise** after attention
• typically expands dimension 4x then projects back
• provides non-linearity and feature transformation. | | [**Residual connections**](https://d2l.ai/chapter_attention-mechanisms-and-transformers/transformer.html#residual-connections-and-layer-normalization) | `output = x + SubLayer(x)` | • Adds input directly to sublayer output enabling **gradient flow** through deep networks
• prevents vanishing gradients and allows training of 100+ layer transformers. | | [**Layer normalization**](https://arxiv.org/abs/1607.06450) | `LN(x) = γ(x - μ) / σ + β` | • Normalizes activations **across feature dimension** for each token
• stabilizes training and enables higher learning rates compared to batch normalization. | ## Table 2: Positional Encoding Variants | **Method** | **Example** | **Description** | |---|---|---| | [**Absolute sinusoidal**](https://kazemnejad.com/blog/transformer_architecture_positional_encoding/) | `PE(pos, 2i) = sin(pos/10000^(2i/d))`
`PE(pos, 2i+1) = cos(pos/10000^(2i/d))` | • Original Transformer encoding using **fixed sine/cosine** functions at different frequencies
• provides unique position signals but doesn't explicitly encode relative distances. | | [**Learned absolute**](https://huggingface.co/docs/transformers/glossary#position-ids) | `pos_emb = Embedding(max_len, d_model)` | • Trainable position embeddings learned during training
• used in BERT and GPT-2
• **limited to max_len** seen during training without extrapolation. | | [**Relative positional encoding**](https://arxiv.org/abs/1803.02155) | `bias = learned_bias[k - q]` | • Encodes **distance between positions** rather than absolute indices
• better length generalization but computationally expensive for long sequences. | | [**RoPE (Rotary Position Embedding)**](https://arxiv.org/abs/2104.09864) | `rotate(q) @ rotate(k).T` | • Applies **rotation matrices** to query-key pairs encoding relative position via complex plane rotation
• used in LLaMA, GPT-NeoX—enables better length extrapolation. | | [**ALiBi (Attention with Linear Biases)**](https://arxiv.org/abs/2108.12409) | `attn + bias * (-1, -2, -3, ...)` | • Adds **linear penalty** to attention scores based on distance
• no position embeddings needed
• excellent extrapolation to longer sequences than training context. | ## Table 3: Model Architecture Variants | **Type** | **Example** | **Description** | |---|---|---| | [**Encoder-only**](https://huggingface.co/docs/transformers/model_doc/bert) | `BERT, RoBERTa` | • Bidirectional attention processes entire sequence
• excels at **understanding tasks** like classification, NER, question answering
• cannot generate text autoregressively. | | [**Decoder-only**](https://huggingface.co/docs/transformers/model_doc/gpt2) | `GPT-2, GPT-3, LLaMA` | • Causal masked attention for **autoregressive generation**
• trained to predict next token
• dominates modern LLMs due to scalability and generation quality. | | [**Encoder-decoder**](https://huggingface.co/docs/transformers/model_doc/t5) | `T5, BART` | • Separate encoder (bidirectional) and decoder (causal) with **cross-attention** layer
• optimal for seq2seq tasks like translation, summarization
• more parameter-efficient for conditional generation. | | [**Mixture of Experts (MoE)**](https://arxiv.org/abs/1701.06538) | `router(x) → top-k experts` | • Sparsely activated architecture where **gating network** routes tokens to subset of expert FFNs
• scales parameters without proportional compute increase (Mixtral, Switch Transformer). | | [**Vision-language**](https://openai.com/research/clip) | `CLIP, LLaVA` | • Integrates **vision encoder** (ViT) with language model via projection layer or cross-attention
• enables multimodal understanding from images and text jointly. | ## Table 4: Tokenization Algorithms | **Algorithm** | **Example** | **Description** | |---|---|---| | [**Byte-Pair Encoding (BPE)**](https://arxiv.org/abs/1508.07909) | `"playing" → ["play", "ing"]` | • Iteratively merges **most frequent character pairs** in training corpus
• balances vocabulary size with coverage
• used in GPT-2, GPT-3—stores merge rules. | | [**WordPiece**](https://arxiv.org/abs/1609.08144) | `"unaffable" → ["un", "##aff", "##able"]` | • Similar to BPE but selects merges based on **likelihood maximization** rather than frequency
• used in BERT
• saves final vocabulary only, not merge operations. | | [**SentencePiece**](https://arxiv.org/abs/1808.06226) | `treats spaces as token "_"` | • Language-agnostic tokenizer operating on **raw text** without pre-tokenization
• encodes whitespace as special character
• supports BPE and unigram models
• used in T5, LLaMA. | | [**Unigram language model**](https://arxiv.org/abs/1804.10959) | `probabilistic token selection` | • Maintains vocabulary with **token probabilities**
• removes tokens iteratively to minimize loss
• allows multiple segmentations unlike greedy BPE/WordPiece. | ## Table 5: Pre-training Objectives | **Objective** | **Example** | **Description** | |---|---|---| | [**Causal language modeling (CLM)**](https://huggingface.co/docs/transformers/tasks/language_modeling) | `P(token_i | token_• standard decoder-only objective maximizing likelihood of training sequences
• used in GPT family. | | [**Masked language modeling (MLM)**](https://arxiv.org/abs/1810.04805) | `P([MASK] | context)` | • Randomly masks ~15% of tokens and predicts them from **bidirectional context**
• BERT's core pre-training objective enabling deep bidirectional representations. | | [**Denoising autoencoding**](https://arxiv.org/abs/1910.13461) | `corrupt → reconstruct` | • Masks or corrupts spans of text then trains model to **reconstruct original**
• T5 frames all tasks as text-to-text generation with varying corruption strategies. | | [**Prefix language modeling**](https://arxiv.org/abs/2105.13626) | `bidirectional prefix → causal` | • Applies bidirectional attention to prefix then **causal attention** for continuation
• bridges encoder and decoder benefits (UniLM, GLM). | | [**Contrastive learning**](https://arxiv.org/abs/2103.00020) | `align(text, image)` | • Trains model to **match positive pairs** (text-image) while separating negative pairs
• CLIP uses dual encoders with contrastive loss for vision-language alignment. | ## Table 6: Fine-Tuning Techniques | **Technique** | **Example** | **Description** | |---|---|---| | [**Supervised fine-tuning (SFT)**](https://huggingface.co/docs/transformers/training) | `train on (input, output) pairs` | • Standard gradient-based training on **task-specific labeled data**
• updates all parameters to adapt pre-trained model to downstream task. | | [**Instruction tuning**](https://arxiv.org/abs/2109.01652) | `"Translate: [text]" → output` | • Fine-tunes on diverse instruction-following datasets formatted as **explicit commands**
• dramatically improves zero-shot task generalization (FLAN, InstructGPT). | | [**LoRA (Low-Rank Adaptation)**](https://arxiv.org/abs/2106.09685) | `ΔW = BA` (rank r << d) | • Freezes base model and trains **low-rank decomposition matrices** injected into attention layers
• reduces trainable parameters by 10,000x while preserving quality. | | [**QLoRA**](https://arxiv.org/abs/2305.14314) | `4-bit base + LoRA adapters` | • Quantizes base model to **4-bit precision** and applies LoRA
• enables fine-tuning 65B models on single 48GB GPU with minimal degradation. | | [**Adapter modules**](https://arxiv.org/abs/1902.00751) | `insert bottleneck layers` | • Adds small trainable **feed-forward modules** between frozen transformer layers
• modular approach allowing task-specific adapters without full model copies. | | [**Prefix tuning**](https://arxiv.org/abs/2101.00190) | `prepend trainable vectors` | • Optimizes continuous **task-specific vectors** prepended to each layer
• keeps model frozen
• competitive with full fine-tuning on some tasks with 0.1% parameters. | | [**P-tuning / Prompt tuning**](https://arxiv.org/abs/2103.10385) | `optimize soft prompts` | • Learns continuous **prompt embeddings** rather than discrete text
• more parameter-efficient than prefix tuning
• effectiveness increases with model scale. | ## Table 7: Alignment and RLHF | **Method** | **Example** | **Description** | |---|---|---| | [**RLHF (Reinforcement Learning from Human Feedback)**](https://arxiv.org/abs/2203.02155) | `reward model → PPO training` | • Three-stage process: train **reward model** on human preferences, optimize policy via RL (PPO) maximizing reward
• used in ChatGPT, Claude—computationally expensive. | | [**DPO (Direct Preference Optimization)**](https://arxiv.org/abs/2305.18290) | `directly optimize preferences` | • Bypasses reward model by **directly optimizing policy** on preference pairs using Bradley-Terry model
• simpler and more stable than RLHF with comparable results. | | [**RLAIF (RL from AI Feedback)**](https://arxiv.org/abs/2309.00267) | `AI-generated preferences` | • Replaces human labelers with **AI-generated feedback** for reward model training
• scales preference data collection
• particularly effective for capability-focused alignment. | | [**Constitutional AI**](https://arxiv.org/abs/2212.08073) | `self-critique + revision` | • Model **critiques own outputs** against constitution principles then revises
• reduces reliance on human feedback for harmlessness alignment. | | [**Reward modeling**](https://arxiv.org/abs/2204.05862) | `score(response) from pairs` | • Trains classifier to predict **human preference** between response pairs
• converts subjective preferences into scalar reward signal for RL optimization. | ## Table 8: Inference Optimization Techniques | **Technique** | **Example** | **Description** | |---|---|---| | [**KV caching**](https://arxiv.org/abs/2211.05102) | `cache computed K, V` | • Stores previously computed **key-value pairs** during autoregressive generation
• avoids redundant computation—essential for production inference efficiency. | | [**Flash Attention**](https://arxiv.org/abs/2205.14135) | `tiling + recomputation` | • IO-aware attention algorithm using **block-wise computation** and kernel fusion
• reduces memory bandwidth by 5-20x enabling 2-4x speedup on long sequences. | | [**PagedAttention**](https://arxiv.org/abs/2309.06180) | `block-level memory management` | • Manages KV cache in **non-contiguous blocks** like OS paging
• reduces memory fragmentation and increases batch size in vLLM by 2-24x. | | [**Speculative decoding**](https://arxiv.org/abs/2211.17192) | `draft model → verify in parallel` | • Small **draft model** generates candidate tokens that large model verifies in parallel
• 2-3x speedup without changing output distribution. | | [**Quantization (INT8/FP8)**](https://arxiv.org/abs/2208.07339) | `convert FP16 → INT8` | • Reduces precision of weights/activations to **8-bit** or lower
• 2-4x memory reduction and speedup with <1% accuracy loss using calibration. | | [**Continuous batching**](https://www.anyscale.com/blog/continuous-batching-llm-inference) | `dynamic request batching` | • Batches requests with **varying sequence lengths** dynamically during generation
• maximizes GPU utilization compared to static batching. | ## Table 9: Sampling and Decoding Strategies | **Strategy** | **Example** | **Description** | |---|---|---| | [**Greedy decoding**](https://huggingface.co/blog/how-to-generate) | `argmax(logits)` | • Selects **highest probability** token at each step
• deterministic and fast but prone to repetitive, low-quality outputs for creative tasks. | | [**Beam search**](https://huggingface.co/blog/how-to-generate) | `keep top-k sequences` | • Maintains **k parallel hypotheses** and expands most probable
• balances quality and diversity but computationally expensive
• common for translation. | | [**Temperature sampling**](https://huggingface.co/blog/how-to-generate) | `logits / T before softmax` | • Scales logits by temperature **T**—lower T (0.1-0.7) makes distribution sharper/conservative
• higher T (1.0-2.0) increases randomness/creativity. | | [**Top-k sampling**](https://arxiv.org/abs/1805.04833) | `sample from top k=40 tokens` | • Restricts sampling to **k highest-probability** tokens
• prevents sampling rare tokens but fixed k can be too restrictive or permissive depending on distribution. | | [**Top-p (nucleus) sampling**](https://arxiv.org/abs/1904.09751) | `cumulative prob >= p=0.9` | • Dynamically selects **smallest set** of tokens whose cumulative probability exceeds threshold p
• adapts to distribution shape—preferred for generation. | | [**Min-p sampling**](https://arxiv.org/abs/2407.01082) | `filter tokens < min_p * max_prob` | • Removes tokens with probability below **min_p fraction** of maximum token probability
• better than top-p for maintaining quality while allowing creativity. | ## Table 10: Training Optimizations | **Technique** | **Example** | **Description** | |---|---|---| | [**Mixed precision (FP16/BF16)**](https://arxiv.org/abs/1710.03740) | `store in FP16, compute in FP32` | • Uses **16-bit floats** for storage/computation with FP32 master weights
• reduces memory by ~2x and accelerates training with Tensor Cores—BF16 preferred for stability. | | [**Gradient accumulation**](https://huggingface.co/docs/transformers/perf_train_gpu_one#gradient-accumulation) | `accumulate over N steps` | • Simulates larger **effective batch size** by accumulating gradients across microbatches before update
• crucial for training large models on limited memory. | | [**Gradient checkpointing**](https://arxiv.org/abs/1604.06174) | `recompute activations` | • Trades compute for memory by **recomputing forward activations** during backward pass instead of storing
• enables 2-10x larger models at 20-30% slowdown. | | [**AdamW optimizer**](https://arxiv.org/abs/1711.05101) | `decouple weight decay` | • Fixes **weight decay in Adam** by applying it directly to weights not gradient
• improves generalization
• standard optimizer for transformer training. | | [**Learning rate warmup**](https://arxiv.org/abs/1706.02677) | `linear increase 0 → max_lr` | • Gradually **increases learning rate** from zero over initial steps (typically 2-10% of training)
• stabilizes training of large models and prevents divergence. | | [**Cosine annealing**](https://arxiv.org/abs/1608.03983) | `lr = min + 0.5(max-min)(1+cos)` | • **Decreases learning rate** following cosine curve
• smooth decay helps model converge to flatter minima
• often combined with warmup and restarts. | ## Table 11: Distributed Training Strategies | **Strategy** | **Example** | **Description** | |---|---|---| | [**Data parallelism**](https://pytorch.org/tutorials/beginner/ddp_series_theory.html) | `replicate model across GPUs` | • Each GPU holds **full model copy** processing different data batches
• gradients averaged across devices
• simplest parallelism but memory-limited by single GPU. | | [**Tensor parallelism**](https://arxiv.org/abs/1909.08053) | `split layers across GPUs` | • Partitions individual layers (e.g., attention heads) **within model** across devices
• requires frequent all-reduce communication
• used in Megatron-LM for very large models. | | [**Pipeline parallelism**](https://arxiv.org/abs/1811.06965) | `layer stages on different GPUs` | • Splits model **vertically by layers** creating pipeline stages
• microbatching reduces bubble overhead
• GPipe, PipeDream frameworks. | | [**Sequence parallelism**](https://arxiv.org/abs/2205.05198) | `partition along sequence dim` | • Splits sequence length across GPUs for **memory-constrained layers** like LayerNorm
• extends tensor parallelism reducing activation memory in long sequences. | | [**ZeRO (Zero Redundancy Optimizer)**](https://arxiv.org/abs/1910.02054) | `shard optimizer states` | • Partitions **optimizer states, gradients, parameters** across devices with on-demand gathering
• ZeRO-3 achieves model parallelism memory efficiency with data parallelism simplicity. | | [**FSDP (Fully Sharded Data Parallel)**](https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/) | `PyTorch native ZeRO-3` | • PyTorch implementation of **ZeRO-3 sharding**
• automatically manages parameter gathering/scattering
• simpler API than DeepSpeed for distributed training. | ## Table 12: Model Compression Techniques | **Technique** | **Example** | **Description** | |---|---|---| | [**Post-training quantization**](https://arxiv.org/abs/2208.07339) | `GPTQ, AWQ` | • Converts trained model weights to **lower precision** (INT4/8) without retraining
• calibration on small dataset
• 3-4x compression with minimal accuracy loss. | | [**Quantization-aware training (QAT)**](https://arxiv.org/abs/2004.09602) | `simulate quantization during training` | • Inserts **fake quantization** operations in forward pass to model precision effects
• typically achieves better accuracy than post-training methods. | | [**Pruning (structured/unstructured)**](https://arxiv.org/abs/2301.00774) | `remove low-magnitude weights` | • Removes unimportant weights or entire **structured components** (heads, layers)
• can reduce parameters 40-60% but requires careful calibration or retraining. | | [**Knowledge distillation**](https://arxiv.org/abs/1503.02531) | `student learns from teacher` | • Trains small **student model** to match outputs or intermediate representations of large teacher
• compresses model while retaining capabilities—DistilBERT example. | | [**Low-rank decomposition**](https://arxiv.org/abs/2106.09685) | `W ≈ UV` (rank r) | • Approximates weight matrices as **product of low-rank matrices**
• reduces parameters in linear layers
• basis of LoRA and similar methods. | ## Table 13: Context Window and Long-Context Techniques | **Technique** | **Example** | **Description** | |---|---|---| | [**Retrieval-Augmented Generation (RAG)**](https://arxiv.org/abs/2005.11401) | `retrieve docs → augment prompt` | • Retrieves relevant documents from external knowledge base and **injects into context**
• extends effective context beyond model limits
• requires good retrieval system. | | [**Sparse attention**](https://arxiv.org/abs/2004.05150) | `attend to subset of positions` | • Computes attention only for **selected position pairs** using patterns (local, strided, global)
• reduces O(n²) complexity enabling 10x+ longer sequences. | | [**Sliding window attention**](https://arxiv.org/abs/2004.05150) | `attend to local window` | • Each token attends only to **fixed-size window** around its position
• linear memory but limited long-range modeling
• used in Longformer. | | [**RoPE scaling**](https://arxiv.org/abs/2306.15595) | `adjust rotation frequencies` | • Modifies RoPE base frequency to **extrapolate to longer sequences**
• simple method enabling 2-4x context extension with minimal fine-tuning. | | [**Position interpolation**](https://arxiv.org/abs/2306.15595) | `compress position indices` | • **Interpolates positions** within training range rather than extrapolating
• better stability than direct extrapolation for extended context. | | [**Recurrent memory**](https://arxiv.org/abs/2404.07143) | `compress past into memory` | • Summarizes earlier context into **compressed memory state**
• enables unbounded context in theory but loses fine-grained information from distant past. | ## Table 14: Prompt Engineering Techniques | **Technique** | **Example** | **Description** | |---|---|---| | [**Zero-shot prompting**](https://arxiv.org/abs/2109.01652) | `"Translate to French: [text]"` | • Provides task instruction only **without examples**
• relies on pre-training and instruction tuning
• quality highly dependent on model capabilities and prompt clarity. | | [**Few-shot prompting**](https://arxiv.org/abs/2005.14165) | `Example1, Example2, ... Query` | • Demonstrates task through **2-10 input-output examples** in prompt
• exploits in-context learning
• effectiveness grows with model scale
• examples should be diverse. | | [**Chain-of-thought (CoT)**](https://arxiv.org/abs/2201.11903) | `"Let's think step by step"` | • Prompts model to generate **intermediate reasoning steps** before final answer
• dramatically improves performance on math, logic, commonsense reasoning. | | [**Self-consistency**](https://arxiv.org/abs/2203.11171) | `sample multiple paths → vote` | • Generates **multiple reasoning paths** then selects most consistent answer via majority voting
• improves reliability over single-path CoT. | | [**Tree-of-thoughts**](https://arxiv.org/abs/2305.10601) | `explore reasoning tree` | • Explores **multiple reasoning branches** with backtracking and evaluation
• enables deliberate problem-solving for complex tasks requiring search. | | [**ReAct (Reasoning + Acting)**](https://arxiv.org/abs/2210.03629) | `Thought → Action → Observation` | • Interleaves **reasoning and tool use**
• model generates thoughts, selects actions (API calls, searches), observes results iteratively until solution found. | ## Table 15: Emergent Capabilities and Scaling | **Concept** | **Example** | **Description** | |---|---|---| | [**Emergent abilities**](https://arxiv.org/abs/2206.07682) | `reasoning, arithmetic` | • Capabilities that appear **suddenly at scale** not present in smaller models
• includes in-context learning, chain-of-thought reasoning
• controversial whether truly emergent or measurement artifacts. | | [**In-context learning**](https://arxiv.org/abs/2005.14165) | `few-shot without gradients` | • Ability to learn new tasks from **examples in prompt** without parameter updates
• improves with scale
• mechanism not fully understood—may involve induction heads. | | [**Scaling laws**](https://arxiv.org/abs/2001.08361) | `L(N) ∝ N^(-α)` | • Loss scales as **power law** with compute, model size, dataset size
• predicts training compute allocation
• vocabulary size also affects optimal scaling (vocab ∝ params^0.83). | | [**Compute-optimal scaling**](https://arxiv.org/abs/2203.15556) | `Chinchilla scaling` | • For fixed compute budget, **balanced scaling** of model size and training tokens is optimal
• suggests many models are undertrained relative to size. | | [**Transfer learning**](https://arxiv.org/abs/1801.06146) | `pre-train → fine-tune` | • Pre-trained models encode **general language understanding** transferable to downstream tasks
• foundation of modern NLP—larger models transfer better. | ## Table 16: Activation Functions in Transformers | **Function** | **Example** | **Description** | |---|---|---| | [**ReLU (Rectified Linear Unit)**](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html) | `ReLU(x) = max(0, x)` | • Simple **piecewise linear** function
• original Transformer used ReLU
• can suffer from dead neurons but computationally efficient
• largely replaced in modern LLMs. | | [**GELU (Gaussian Error Linear Unit)**](https://arxiv.org/abs/1606.08415) | `GELU(x) = x·Φ(x)` | • Smooth approximation applying **Gaussian CDF**
• used in BERT, GPT-2
• better gradient properties than ReLU
• probabilistic interpretation as neuron dropout. | | [**SwiGLU**](https://arxiv.org/abs/2002.05202) | `SwiGLU(x) = Swish(xW) ⊙ (xV)` | • Gated variant using **Swish activation** (x·sigmoid(x)) with element-wise gating
• used in LLaMA, PaLM—empirically outperforms GELU with ~50% more parameters in FFN. | | [**GeGLU**](https://arxiv.org/abs/2002.05202) | `GeGLU(x) = GELU(xW) ⊙ (xV)` | • Similar to SwiGLU but uses **GELU** for gating
• strong performance on language tasks
• T5 variant
• computational overhead mitigated by performance gains. | ## Table 17: Normalization Techniques | **Method** | **Example** | **Description** | |---|---|---| | [**Layer Normalization (LN)**](https://arxiv.org/abs/1607.06450) | `LN(x) = (x - μ) / σ · γ + β` | • Normalizes across **feature dimension** for each token independently
• standard in transformers
• mean/variance computed per-sample allowing any batch size. | | [**RMSNorm (Root Mean Square Normalization)**](https://arxiv.org/abs/1910.07467) | `RMSNorm(x) = x / RMS(x) · γ` | • Simplified LN removing **mean centering**—only normalizes by RMS
• 10-20% faster than LN with comparable performance
• used in LLaMA, Grok. | | [**Pre-LN vs Post-LN**](https://arxiv.org/abs/2002.04745) | `Pre: LN(x) → Sublayer` | • **Pre-LN** applies normalization before sublayer (modern default—more stable)
• **Post-LN** applies after (original Transformer—requires warmup)
• Pre-LN enables easier convergence. | | [**Dropout**](https://jmlr.org/papers/v15/srivastava14a.html) | `randomly zero with p=0.1` | • Randomly **drops activations** during training as regularization
• less common in very large LLMs which are underparameterized relative to data
• typical rates 0.1-0.2. | ## Table 18: Evaluation Metrics and Benchmarks | **Metric** | **Example** | **Description** | |---|---|---| | [**Perplexity**](https://huggingface.co/docs/transformers/perplexity) | `PPL = exp(avg_loss)` | • Measures how **surprised** model is by test data
• lower is better
• exponential of average cross-entropy loss
• common language modeling metric. | | [**BLEU (Bilingual Evaluation Understudy)**](https://aclanthology.org/P02-1040/) | `n-gram precision with brevity` | • Compares **n-gram overlap** between generated and reference translations
• 0-100 scale
• emphasizes precision
• standard for machine translation evaluation. | | [**ROUGE (Recall-Oriented Understudy for Gisting)**](https://aclanthology.org/W04-1013/) | `ROUGE-L, ROUGE-N` | • Measures **recall-oriented n-gram overlap**
• focuses on coverage of reference
• primarily for summarization
• ROUGE-L uses longest common subsequence. | | [**BERTScore**](https://arxiv.org/abs/1904.09675) | `contextual embedding similarity` | • Computes **token similarity** using BERT embeddings rather than exact matches
• captures semantic similarity better than n-gram metrics. | | [**MMLU (Massive Multitask Language Understanding)**](https://arxiv.org/abs/2009.03300) | `57 subjects, 4-way multiple choice` | • Tests **knowledge and reasoning** across STEM, humanities, social sciences
• standard benchmark for evaluating general capabilities
• 0-100% accuracy. | | [**HumanEval**](https://arxiv.org/abs/2107.03374) | `code synthesis benchmark` | • Evaluates **code generation** with 164 hand-written programming problems
• pass@k metric measures functional correctness
• standard for coding models. | ## Table 19: Attention Mechanism Optimizations | **Optimization** | **Example** | **Description** | |---|---|---| | [**Multi-query attention (MQA)**](https://arxiv.org/abs/1911.02150) | `single K, V across heads` | • Shares **same key-value** projections across all heads using only multiple queries
• reduces KV cache memory and speeds inference but slightly lower quality. | | [**Grouped-query attention (GQA)**](https://arxiv.org/abs/2305.13245) | `heads share K, V in groups` | • Compromise between MQA and MHA: **groups of heads** share K, V
• balances quality-efficiency tradeoff
• used in LLaMA-2, Mistral. | | [**Sliding window attention**](https://arxiv.org/abs/2004.05150) | `attend to k nearest tokens` | • Restricts attention to **fixed-size local window**
• reduces complexity to O(n·k) from O(n²)
• enables longer sequences but limited global context. | | [**Flash Attention 2/3**](https://arxiv.org/abs/2307.08691) | `improved kernel fusion` | • Enhanced IO-aware kernels with **lower SRAM usage** and better parallelization
• Flash-3 up to 2x faster than Flash-2 on H100 with asynchrony optimizations. | | [**Linear attention**](https://arxiv.org/abs/2006.16236) | `kernel trick approximation` | • Approximates softmax attention using **kernel methods** reducing complexity to O(n)
• enables efficient very long sequences but quality gaps remain. | ## Table 20: Advanced Training Techniques | **Technique** | **Example** | **Description** | |---|---|---| | [**Curriculum learning**](https://arxiv.org/abs/2101.10382) | `easy → hard examples` | • Orders training data from **simple to complex**
• can improve convergence and final performance
• domain-specific curriculum design needed. | | [**Data augmentation**](https://arxiv.org/abs/2110.01852) | `backtranslation, paraphrasing` | • Generates **synthetic training variations** from existing data
• back-translation, EDA, GPT-generated examples
• particularly useful for low-resource tasks. | | [**Contrastive learning**](https://arxiv.org/abs/2004.11362) | `SimCLR, CLIP` | • Learns representations by **contrasting positive pairs** against negatives
• CLIP aligns text-image pairs
• effective for self-supervised and multimodal learning. | | [**Multi-task learning**](https://arxiv.org/abs/1910.10683) | `train on multiple tasks jointly` | • Shares parameters across tasks expecting **positive transfer**
• T5 frames everything as text-to-text
• requires balanced sampling and task weighting. | | [**Continual learning**](https://arxiv.org/abs/2302.00487) | `incremental data updates` | • Updates model on **new data without forgetting** previous knowledge
• addresses catastrophic forgetting through rehearsal, regularization, or architectural solutions. | ---

Large Language Models (LLMs) Cheat Sheet

Large Language Models (LLMs) Cheat Sheet

Large Language Models (LLMs) Cheat Sheet

Large Language Models (LLMs) Cheat Sheet