Transformer Architecture Cheat Sheet

Updated 2026-05-25

Next Topic: Variational Autoencoders (VAEs) Cheat Sheet

🧠Study flashcards on this topic115 cards · spaced repetition→

Transformer architecture, introduced in 2017 by Vaswani et al., revolutionized deep learning by replacing recurrent and convolutional layers with an attention-based mechanism that processes sequences in parallel. Unlike RNNs and LSTMs, transformers eliminate sequential dependencies through self-attention, allowing every token to attend to every other token simultaneously — enabling massively parallel training that scales to hundreds of billions of parameters. At its core, the transformer relies on three key concepts: multi-head attention for capturing diverse relationships, positional encodings for sequence order awareness, and residual connections with layer normalization for stable deep network training. Understanding transformers is essential not only because they power modern large language models like GPT, BERT, LLaMA, and DeepSeek, but also because their architectural innovations — from encoder-decoder structures to efficient attention variants and hybrid architectures — have influenced every domain in AI, from vision (ViT) to multimodal and agentic systems.

What This Cheat Sheet Covers

This topic spans 20 focused tables and 115 indexed concepts, 115 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Core Attention MechanismsTable 2: Query, Key, Value ComputationsTable 3: Positional Encoding TechniquesTable 4: Layer Normalization and Residual ConnectionsTable 5: Feed-Forward NetworkTable 6: Encoder and Decoder ArchitecturesTable 7: Transformer Variants and ModelsTable 8: Model HyperparametersTable 9: Training TechniquesTable 10: Optimization AlgorithmsTable 11: Training Paradigms and ObjectivesTable 12: Post-Training AlignmentTable 13: Parameter-Efficient Fine-Tuning (PEFT)Table 14: Token Embeddings and VocabularyTable 15: Inference and Decoding StrategiesTable 16: Attention Complexity and EfficiencyTable 17: Inference Serving and OptimizationTable 18: Model QuantizationTable 19: Advanced Architectural ConceptsTable 20: Alternative Architectures

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Core Attention Mechanisms

Self-attention is the engine of every transformer block: it lets each token build a context-aware representation by attending to all other tokens simultaneously. The many variants below trade off memory, speed, and quality differently — choosing the right one is critical for modern LLM design.

Mechanism	Example	Description
Self-Attention	`Attention(Q, K, V) = softmax(QK^T / sqrt(d_k))V`	• Allows each position to attend to all positions in the same sequence • query, key, and value all derived from the same input • forms the foundation of every transformer encoder and decoder block
Scaled Dot-Product Attention	`score = (Q @ K.T) / sqrt(d_k)` `attn_weights = softmax(score)`	• Computes attention scores via dot product of queries and keys, scaled by $\frac{1}{\sqrt{d_k}}$ to prevent gradient saturation in softmax • scaling maintains stable variance as embedding dimensions grow
Multi-Head Attention	8 heads with $d_{model}=512$ `head_dim = 512/8 = 64`	• Splits Q, K, V into h parallel attention heads each of dimension $d_k = d_{model}/h$ • enables the model to jointly attend to information from different representation subspaces • outputs concatenated and linearly projected
Cross-Attention	Decoder attends to encoder: `Q=decoder, K=encoder, V=encoder`	• Queries come from one sequence (decoder) while keys and values come from another (encoder) • allows decoder to focus on relevant encoder positions • essential in encoder-decoder architectures for seq2seq tasks
Masked Self-Attention	Upper-triangular mask: `[[0,-inf,-inf],[0,0,-inf],[0,0,0]]`	• Prevents positions from attending to subsequent positions by applying a causal mask before softmax • ensures token $i$ only sees tokens $\leq i$ • used in decoder-only models (GPT) and decoder blocks
Grouped-Query Attention (GQA)	32 heads in 8 groups (4 heads/group)	• Groups multiple query heads to share key-value pairs • balances memory efficiency and model quality • achieves 99%+ of MHA quality with 2-3× faster inference • used in LLaMA 2/3, Mistral, Gemini
Multi-Query Attention (MQA)	8 query heads share 1 K-V pair	• Uses single shared key and value across all query heads • drastically reduces KV cache memory • trades slight quality loss for up to 2-5× faster decoding.
Multi-Head Latent Attention (MLA)	Cache compressed latent $c_{KV}$ then decompress to K, V per head	• Replaces K, V projections with low-rank factorization: compresses to a latent vector, caches it, decompresses on the fly • reduces KV cache to ~5-13% of MHA while maintaining full per-head expressiveness • includes decoupled RoPE for position-aware heads • used in DeepSeek-V2/V3.
Native Sparse Attention (NSA)	Hierarchical: compressed blocks + selected important tokens + sliding window	• Hardware-aligned natively trainable sparse attention combining coarse-grained token compression, fine-grained selective token attention, and a local sliding-window • enables efficient long-context training (128K+) with hardware-friendly sparsity • proposed by DeepSeek (2025).

Table 2: Query, Key, Value Computations

The QKV mechanism is how raw input embeddings become the query, key, and value representations that drive attention. Understanding this linear transformation layer is essential for grasping what attention actually learns.

Component	Example	Description
Q, K, V Projection Matrices	`Q = X @ W_Q` `K = X @ W_K` `V = X @ W_V`	• Learned linear transformations applied to input embeddings • weight matrices $W_Q, W_K, W_V \in \mathbb{R}^{d_{model} \times d_k}$ trained via backpropagation • each attention head has its own projection matrices

The rest of this section is part of the full cheat sheet

Table 3: Positional Encoding Techniques

Transformers have no inherent notion of order — positional encodings inject sequence position information into token embeddings. The right encoding scheme determines whether a model can generalize to longer sequences than seen during training.

Technique	Example	Description
Sinusoidal Positional Encoding	$PE_{(pos,2i)} = \sin(pos/10000^{2i/d})$ $PE_{(pos,2i+1)} = \cos(pos/10000^{2i/d})$	• Fixed encoding using sine and cosine at different frequencies • allows extrapolation to longer sequences • position differences correspond to linear transformations in embedding space • used in original Transformer

The rest of this section is part of the full cheat sheet

Table 4: Layer Normalization and Residual Connections

Normalization and skip connections are what make training a 96-layer transformer stable. Without them, gradients either vanish or explode before reaching the early layers.

Component	Example	Description
Residual Connections	`output = x + SubLayer(x)`	• Adds input directly to sub-layer output via skip connection • enables gradient flow through deep networks • mitigates vanishing gradients • allows each layer to learn refinements rather than full transformations • critical for training 12-96+ layer transformers

The rest of this section is part of the full cheat sheet

Table 1: Core Attention Mechanisms

Mechanism	Example	Description
Self-Attention	`Attention(Q, K, V) = softmax(QK^T / sqrt(d_k))V`	• Allows each position to attend to all positions in the same sequence • query, key, and value all derived from the same input • forms the foundation of every transformer encoder and decoder block
Scaled Dot-Product Attention	`score = (Q @ K.T) / sqrt(d_k)` `attn_weights = softmax(score)`	• Computes attention scores via dot product of queries and keys, scaled by $\frac{1}{\sqrt{d_k}}$ to prevent gradient saturation in softmax • scaling maintains stable variance as embedding dimensions grow
Multi-Head Attention	8 heads with $d_{model}=512$ `head_dim = 512/8 = 64`	• Splits Q, K, V into h parallel attention heads each of dimension $d_k = d_{model}/h$ • enables the model to jointly attend to information from different representation subspaces • outputs concatenated and linearly projected
Cross-Attention	Decoder attends to encoder: `Q=decoder, K=encoder, V=encoder`	• Queries come from one sequence (decoder) while keys and values come from another (encoder) • allows decoder to focus on relevant encoder positions • essential in encoder-decoder architectures for seq2seq tasks
Masked Self-Attention	Upper-triangular mask: `[[0,-inf,-inf],[0,0,-inf],[0,0,0]]`	• Prevents positions from attending to subsequent positions by applying a causal mask before softmax • ensures token $i$ only sees tokens $\leq i$ • used in decoder-only models (GPT) and decoder blocks
Grouped-Query Attention (GQA)	32 heads in 8 groups (4 heads/group)	• Groups multiple query heads to share key-value pairs • balances memory efficiency and model quality • achieves 99%+ of MHA quality with 2-3× faster inference • used in LLaMA 2/3, Mistral, Gemini
Multi-Query Attention (MQA)	8 query heads share 1 K-V pair	• Uses single shared key and value across all query heads • drastically reduces KV cache memory • trades slight quality loss for up to 2-5× faster decoding.
Multi-Head Latent Attention (MLA)	Cache compressed latent $c_{KV}$ then decompress to K, V per head	• Replaces K, V projections with low-rank factorization: compresses to a latent vector, caches it, decompresses on the fly • reduces KV cache to ~5-13% of MHA while maintaining full per-head expressiveness • includes decoupled RoPE for position-aware heads • used in DeepSeek-V2/V3.
Native Sparse Attention (NSA)	Hierarchical: compressed blocks + selected important tokens + sliding window	• Hardware-aligned natively trainable sparse attention combining coarse-grained token compression, fine-grained selective token attention, and a local sliding-window • enables efficient long-context training (128K+) with hardware-friendly sparsity • proposed by DeepSeek (2025).

Table 2: Query, Key, Value Computations

Component	Example	Description
Q, K, V Projection Matrices	`Q = X @ W_Q` `K = X @ W_K` `V = X @ W_V`	• Learned linear transformations applied to input embeddings • weight matrices $W_Q, W_K, W_V \in \mathbb{R}^{d_{model} \times d_k}$ trained via backpropagation • each attention head has its own projection matrices

Table 3: Positional Encoding Techniques

Technique	Example	Description
Sinusoidal Positional Encoding	$PE_{(pos,2i)} = \sin(pos/10000^{2i/d})$ $PE_{(pos,2i+1)} = \cos(pos/10000^{2i/d})$	• Fixed encoding using sine and cosine at different frequencies • allows extrapolation to longer sequences • position differences correspond to linear transformations in embedding space • used in original Transformer

Table 4: Layer Normalization and Residual Connections

Normalization and skip connections are what make training a 96-layer transformer stable. Without them, gradients either vanish or explode before reaching the early layers.

Component	Example	Description
Residual Connections	`output = x + SubLayer(x)`	• Adds input directly to sub-layer output via skip connection • enables gradient flow through deep networks • mitigates vanishing gradients • allows each layer to learn refinements rather than full transformations • critical for training 12-96+ layer transformers