Large Language Models (LLMs) Cheat Sheet

Table 1: Core Transformer Architecture

Every modern LLM is built from this handful of components stacked dozens of times over. Self-attention is the heart of it — letting each token weigh every other token — and the rest exist to make that work at depth and scale: feed-forward layers add non-linear capacity, residual connections and normalization keep gradients flowing through hundreds of layers, and positional encoding restores the word-order information that attention alone throws away. Master these and you understand the skeleton shared by GPT, LLaMA, and Qwen alike.

Table 2: Positional Encoding Variants

Since attention treats a sequence as an unordered bag of tokens, a model needs some way to know which word came first — and the choice of how to inject that order has outsized consequences for how far a model can extrapolate beyond its training length. These variants trace the field's evolution from the original fixed sinusoids and learned absolute embeddings toward relative schemes like RoPE and ALiBi that generalize to longer contexts, and even NoPE, which drops explicit position signals entirely.

Table 3: Model Architecture Variants

The same transformer building blocks can be wired in fundamentally different ways, and that wiring decides what a model is good at. Decoder-only models dominate today because their autoregressive design scales and generates so well, but encoder-only models still win on pure understanding tasks, and encoder-decoder models suit translation. The later rows cover the structural innovations reshaping frontier models — Mixture of Experts for scaling parameters cheaply, reasoning models that think before answering, and vision-language models that fold images into the same context.

Table 4: Tokenization Algorithms

Before a model ever sees text it has to be chopped into tokens, and the algorithm that does the chopping shapes vocabulary size, how gracefully the model handles rare or unseen words, and even how well it works across languages. These methods mostly differ in how they decide which character or subword pieces to merge — by frequency in BPE, by likelihood in WordPiece, by probability in Unigram — with byte-level variants guaranteeing that any string at all can be encoded without unknown tokens.

Table 5: Pre-training Objectives

The pre-training objective is the self-supervised game a model plays over trillions of tokens, and it determines what kind of model you end up with. Predicting the next token (CLM) gives you a generator like GPT; masking and predicting tokens from both sides (MLM) gives you a bidirectional understander like BERT; corruption-and-reconstruction objectives power text-to-text models like T5. The remaining rows cover newer twists like multi-token prediction for speed and contrastive learning for aligning text with images.

Table 6: Fine-Tuning Techniques

Once a model is pre-trained, fine-tuning adapts it to a specific task or behavior — and the central tension here is cost versus completeness. Full supervised fine-tuning updates every parameter, but the parameter-efficient methods that follow (LoRA, QLoRA, DoRA, adapters, prefix and prompt tuning) freeze the base model and train a tiny fraction of new weights instead, slashing memory enough to fine-tune huge models on a single GPU while keeping quality close. Instruction tuning sits apart as the step that teaches a model to follow commands rather than just continue text.

Table 7: Alignment and RLHF

Alignment is what turns a raw next-token predictor into a model that's helpful, harmless, and actually does what you ask, and this is one of the fastest-moving areas in the field. The lineage runs from classic three-stage RLHF (reward model plus PPO) toward simpler, cheaper successors — DPO and its variants skip the separate reward model, GRPO drops the critic, and RLVR replaces human labels with programmatic verifiers, which is precisely what unlocked the self-reflecting reasoning models like DeepSeek-R1.

Table 8: Inference Optimization Techniques

Serving an LLM cheaply and fast is its own engineering discipline, and these techniques attack the bottlenecks that make autoregressive generation slow and memory-hungry. KV and prefix caching avoid recomputing work across tokens and requests, Flash Attention and PagedAttention squeeze far more out of GPU memory, and speculative methods like draft models and Medusa generate several tokens at once to break the one-token-at-a-time barrier. Together they're what make production inference economically viable.

Table 9: Sampling and Decoding Strategies

A model outputs a probability distribution over the next token; the decoding strategy decides which token actually gets picked, and it's the single biggest lever on whether output feels robotic or creative. Greedy decoding always grabs the top choice and tends toward repetition, while temperature, top-p, top-k, and min-p sampling inject controlled randomness to keep generation lively without going off the rails. Beam and contrastive search take different routes for tasks like translation that reward coherence over surprise.

Table 10: Training Optimizations

Training a large model bumps into the hard limits of GPU memory and time, and these techniques are the everyday tricks for staying within them. Mixed precision and gradient checkpointing trade bits and recomputation for room to fit bigger models, gradient accumulation fakes a larger batch size, and the optimizer-and-schedule choices — AdamW with warmup and cosine decay — are the near-universal recipe that keeps large-model training from diverging.

Table 11: Distributed Training Strategies

No single GPU can hold a frontier model, so training is spread across many — and these strategies differ in what they split. Data parallelism replicates the model and splits the batch; tensor and pipeline parallelism slice the model itself, across layers or within them; and ZeRO/FSDP shard the optimizer states and parameters to wring out memory. Real large-scale runs combine several of these at once, so understanding what each one partitions is the key to reading any training setup.

Table 12: Model Compression Techniques

Compression shrinks a trained model so it runs faster and cheaper, ideally without losing much accuracy. Quantization drops weights to lower precision, distillation trains a small student to imitate a large teacher, and pruning cuts out weights or whole components that contribute little — each making a different trade between how much you save and how much retraining or calibration it costs.

Table 13: Context Window and Long-Context Techniques

A model can only attend over so many tokens at once, and attention's quadratic cost makes simply growing that window expensive — so these techniques extend a model's effective reach in two complementary ways. RAG sidesteps the limit by retrieving relevant text on demand, while RoPE scaling, position interpolation, and YaRN stretch the position encoding to handle far longer sequences than training; sliding-window and sparse attention cut the quadratic cost so the longer context is actually affordable to run.

Table 14: Prompt Engineering Techniques

Prompting is how you steer a frozen model at inference time, no training required — and a surprising amount of capability is unlocked just by how you ask. Few-shot and zero-shot prompting set the baseline, while the reasoning-oriented techniques are the real workhorses: chain-of-thought coaxes the model to show its steps, self-consistency votes across multiple attempts, and ReAct and tree-of-thoughts add tool use and search for genuinely hard problems.

Table 15: Emergent Capabilities and Scaling

This is the theory behind why making models bigger keeps working. Scaling laws predict how loss falls as a power law with compute, data, and size, and the Chinchilla result refined that into how to balance the budget — revealing that many models were badly undertrained. The more surprising entries are in-context learning and emergent abilities: capabilities like few-shot learning and reasoning that appear seemingly out of nowhere once a model crosses a certain scale.

Table 16: Activation Functions in Transformers

The activation function inside each feed-forward layer is a small choice with measurable effect on quality. The field has drifted from the simple ReLU of the original Transformer toward smoother and gated variants — GELU in BERT and GPT-2, and the gated SwiGLU and GeGLU that power LLaMA and PaLM because they consistently squeeze out a bit more performance, at the cost of slightly larger layers.

Table 17: Normalization Techniques

Normalization keeps activations in a stable range so deep transformers can train without blowing up, and two questions dominate the design: which normalizer, and where to put it. RMSNorm has largely displaced classic LayerNorm in modern LLMs for being faster at comparable quality, while the Pre-LN versus Post-LN placement decides how stable training is and whether you need a warmup. Dropout, once standard, fades in the largest models that are data-starved rather than overfit.

Table 18: Evaluation Metrics and Benchmarks

Knowing how good a model actually is means picking the right yardstick, and these fall into two camps. Benchmarks like MMLU, GPQA, HumanEval, and SWE-bench probe knowledge, science, and coding ability — though many are nearing saturation or risk contamination, which is why contamination-resistant and human-preference evaluations (LiveBench, Chatbot Arena) have gained ground. The classic automatic metrics — perplexity, BLEU, ROUGE, BERTScore — measure narrower properties like fluency, translation overlap, and summarization quality.

Table 19: Attention Mechanism Optimizations

Standard multi-head attention is expensive at inference, mostly because of the memory the KV cache consumes, so these optimizations rework attention to be cheaper. GQA and MQA shrink the cache by sharing keys and values across heads, MLA compresses them into a low-rank latent (the trick behind DeepSeek-V3), and sliding-window, linear, and Flash Attention attack the quadratic cost from different angles — most of them trading a sliver of quality for major savings in speed and memory.

Table 20: Advanced Training Techniques

Beyond the standard recipe, these techniques shape what and in what order a model learns. Multi-task and curriculum learning structure the training signal — sharing parameters across tasks, or sequencing easy examples before hard ones — while continual learning tackles the problem of updating a model on new data without erasing what it already knew. Contrastive learning and data augmentation round out the toolkit for representation quality and squeezing more from limited data.

Table 21: Model Merging Techniques

Merging combines two or more fine-tuned models into one without any retraining, blending their abilities just by manipulating weights — a cheap way to fuse, say, a coding model and a chat model. The methods differ in how they reconcile conflicting weights: SLERP interpolates two models geometrically, task arithmetic adds and subtracts capability "vectors," and TIES and DARE resolve interference when merging many models at once, with evolutionary search automating the recipe hunt.

Table 22: LLM Agent Concepts

An agent is what you get when an LLM can act in the world rather than just talk — and tool use, the ability to call external functions through structured JSON, is the capability that crosses that line. The rest describe how agents string those actions together: the ReAct loop interleaves reasoning with acting, memory spans a single session or persists across many, planning decomposes big tasks, and multi-agent setups and RLVR-trained agents push toward systems that solve genuinely multi-step problems on their own.