Deep Learning Cheat Sheet

Updated 2026-08-01

🎯Take a practice test on this topic7 practice tests · 258 questions→

🧠Study flashcards on this topic211 cards · spaced repetition→

Deep Learning is a subset of machine learning that uses multi-layered artificial neural networks to learn hierarchical representations of data, enabling computers to automatically discover patterns and features without explicit programming. It powers modern AI applications from computer vision and natural language processing to speech recognition and autonomous systems. The field has evolved from basic feedforward networks to sophisticated architectures like transformers, state space models, and diffusion models, with gradient-based optimization and backpropagation remaining the fundamental learning mechanisms. Understanding the trade-offs between model capacity, computational efficiency, and generalization is critical for practitioners building production systems.

What This Cheat Sheet Covers

This topic spans 21 focused tables and 232 indexed concepts, 211 flashcards, 7 practice tests with 258 questions. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Neural Network ArchitecturesTable 2: Layer TypesTable 3: Activation FunctionsTable 4: Optimization AlgorithmsTable 5: Loss FunctionsTable 6: Regularization TechniquesTable 7: Attention MechanismsTable 8: Embedding TechniquesTable 9: Training TechniquesTable 10: Initialization MethodsTable 11: Model Compression TechniquesTable 12: Evaluation MetricsTable 13: Common Pitfalls and SolutionsTable 14: Advanced ArchitecturesTable 15: Specialized Convolution OperationsTable 16: Residual Connections and Skip ConnectionsTable 17: Normalization TechniquesTable 18: Learning Rate SchedulesTable 19: Batch Processing ConceptsTable 20: Distributed Training TechniquesTable 21: Deep Learning Frameworks

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Neural Network Architectures

These are the blueprints: the high-level shapes a network can take, each suited to a different kind of data. Convolutional networks and their descendants (ResNet, VGG, Inception) dominate images, recurrent designs (RNN, LSTM, GRU) handle sequences, and transformers and Vision Transformers have largely taken over both, while GANs, VAEs, and autoencoders specialize in generating and compressing data.

Architecture	Example	Description
Transformer	`layer = nn.TransformerEncoderLayer(` `d_model=512, nhead=8)` `encoder = nn.TransformerEncoder(layer, num_layers=6)`	• Attention-based architecture processing sequences in parallel, with no recurrence • dominant in NLP, vision, and multimodal AI, though self-attention cost grows quadratically with sequence length.
Convolutional Neural Network (CNN)	`model = Sequential([` `Conv2D(32, 3, activation='relu'),` `MaxPooling2D(2),` `Flatten(), Dense(10)])`	• Grid-structured data processor using convolutional layers for spatial feature extraction • learns a hierarchy of features, edges and colours early, whole objects deeper.
Long Short-Term Memory (LSTM)	`model = Sequential([` `LSTM(128, return_sequences=True),` `Dense(output_dim)])`	• RNN variant with three gates (input, forget, output) guarding an additive cell state • mitigates the vanishing gradient problem for long-term dependencies (it does not eliminate it).
ResNet (Residual Network)	`x = Conv2D(64, 3)(x)` `x = Add()([x, shortcut])` `x = Activation('relu')(x)`	• Deep CNN with skip connections that add a block's input to its output • made 100+ layer networks trainable, reaching 152 layers at lower complexity than VGG.
Vision Transformer (ViT)	`model = VisionTransformer(` `image_size=224, patch_size=16, num_classes=1000)`	• Applies transformer architecture to image patches treated as tokens • lacks a CNN's built-in locality and translation equivariance, so it needs large-scale pretraining to match one.
Generative Adversarial Network (GAN)	`generator = Sequential([Dense(256), ...])` `discriminator = Sequential([Dense(256), ...])`	• Dual-network system with generator creating samples and discriminator distinguishing real from fake • the two losses are adversarial, so a winning discriminator starves the generator of gradient.
Variational Autoencoder (VAE)	`encoder = Sequential([Dense(latent_dim*2)])` `decoder = Sequential([Dense(input_dim)])`	• Probabilistic generative model learning latent distributions you can sample from • enables controlled generation, though image samples tend to be blurrier than a GAN's.
Autoencoder	`encoder = Sequential([Dense(128), Dense(64)])` `decoder = Sequential([Dense(128), Dense(input_dim)])`	• Unsupervised network trained to copy its input to its output through a bottleneck • used for dimensionality reduction, denoising, and feature learning.
U-Net	`encoder_output = Conv2D(...)(x)` `decoder_input = Concatenate()([upsampled, encoder_output])`	• CNN with a contracting encoder-decoder structure plus skip connections • outputs a per-pixel map at input resolution for image segmentation.
Recurrent Neural Network (RNN)	`model = Sequential([` `SimpleRNN(128, return_sequences=True),` `Dense(vocab_size)])`	• Sequential data processor with hidden state that carries temporal information • cannot parallelize across timesteps, and suffers vanishing gradients on long sequences.
Gated Recurrent Unit (GRU)	`model = Sequential([` `GRU(128, return_sequences=True),` `Dense(output_dim)])`	• Simplified LSTM with two gates (update, reset) and no separate cell state • fewer parameters and faster to compute, with comparable performance.
Bidirectional RNN	`model = Sequential([` `Bidirectional(LSTM(128)),` `Dense(output_dim)])`	• RNN processing sequences in both forward and backward directions • needs the whole sequence up front, so it cannot be used for streaming or causal generation.
AlexNet	`model = Sequential([` `Conv2D(96, 11, strides=4), MaxPooling2D(3, 2),` `Conv2D(256, 5), Flatten(), Dense(4096)])`	• 8-layer GPU-trained CNN that revolutionized ImageNet in 2012 • used ReLU and dropout to show learned features beat hand-designed ones.
VGG	`model = Sequential([` `Conv2D(64, 3), Conv2D(64, 3), MaxPooling2D(),` `Conv2D(128, 3), Conv2D(128, 3)])`	• Deep CNN using uniform 3×3 convolutions stacked in repeating blocks, 16-19 weight layers • two stacked 3×3 convs match one 5×5's receptive field with fewer parameters.
Inception (GoogLeNet)	`x = Concatenate()([` `Conv2D(64, 1)(x), Conv2D(128, 3)(x),` `Conv2D(32, 5)(x), MaxPooling2D(3)(x)])`	• Multi-scale CNN with four parallel branches of different kernel sizes • concatenates them so one block extracts features at several scales at once.

Table 2: Layer Types

Architectures are assembled from layers, and each layer type does one specific job. Dense, convolutional, recurrent, and attention layers do the actual learning; pooling and flattening only reshape what flows through, holding no weights of their own; and normalization and dropout layers keep training stable. Knowing what each one computes, what its knobs do, and what shape comes out the other side is how you read and build any network.

Type	Example	Description
Dense (Fully Connected)	`Dense(128, activation='relu')`	• Connects every input neuron to every output neuron • parameters = inputs × outputs plus biases, which is why a dense head usually dominates a CNN's parameter count.

The rest of this section is part of the full cheat sheet

Table 3: Activation Functions

Activation functions are where a network's nonlinearity comes from. Leave them out and every stacked layer collapses back into a single linear map, so depth buys you nothing at all. Which one you pick shapes how gradients behave: ReLU and its variants made deep training far more practical, while GELU, Swish and the gated SwiGLU became the quiet default inside modern transformers and LLMs.

Function	Example	Description
ReLU (Rectified Linear Unit)	`Activation('relu')` `# f(x) = max(0, x)`	• Outputs the input if positive, zero otherwise • cheap to compute and non-saturating for positives, which greatly speeds up convergence.

The rest of this section is part of the full cheat sheet

Table 4: Optimization Algorithms

The optimizer decides how the network turns each batch of gradients into an actual weight update, and that choice shapes both how fast you train and how well the result generalizes. Most of these differ in just two decisions: whether they smooth the gradient with momentum, and whether they give each parameter its own step size. Adam and AdamW are the everyday defaults, while memory-frugal options like Lion and Adafactor exist because at large-model scale the optimizer's own state rivals the size of the model.

Optimizer	Example	Description
Adam (Adaptive Moment Estimation)	`Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999)`	• Tracks two running averages per weight: a smoothed gradient (momentum) and recent squared gradients (per-parameter step size) • both start at zero, so bias correction rescales them during early steps.

The rest of this section is part of the full cheat sheet

Table 1: Neural Network Architectures

Architecture	Example	Description
Transformer	`layer = nn.TransformerEncoderLayer(` `d_model=512, nhead=8)` `encoder = nn.TransformerEncoder(layer, num_layers=6)`	• Attention-based architecture processing sequences in parallel, with no recurrence • dominant in NLP, vision, and multimodal AI, though self-attention cost grows quadratically with sequence length.
Convolutional Neural Network (CNN)	`model = Sequential([` `Conv2D(32, 3, activation='relu'),` `MaxPooling2D(2),` `Flatten(), Dense(10)])`	• Grid-structured data processor using convolutional layers for spatial feature extraction • learns a hierarchy of features, edges and colours early, whole objects deeper.
Long Short-Term Memory (LSTM)	`model = Sequential([` `LSTM(128, return_sequences=True),` `Dense(output_dim)])`	• RNN variant with three gates (input, forget, output) guarding an additive cell state • mitigates the vanishing gradient problem for long-term dependencies (it does not eliminate it).
ResNet (Residual Network)	`x = Conv2D(64, 3)(x)` `x = Add()([x, shortcut])` `x = Activation('relu')(x)`	• Deep CNN with skip connections that add a block's input to its output • made 100+ layer networks trainable, reaching 152 layers at lower complexity than VGG.
Vision Transformer (ViT)	`model = VisionTransformer(` `image_size=224, patch_size=16, num_classes=1000)`	• Applies transformer architecture to image patches treated as tokens • lacks a CNN's built-in locality and translation equivariance, so it needs large-scale pretraining to match one.
Generative Adversarial Network (GAN)	`generator = Sequential([Dense(256), ...])` `discriminator = Sequential([Dense(256), ...])`	• Dual-network system with generator creating samples and discriminator distinguishing real from fake • the two losses are adversarial, so a winning discriminator starves the generator of gradient.
Variational Autoencoder (VAE)	`encoder = Sequential([Dense(latent_dim*2)])` `decoder = Sequential([Dense(input_dim)])`	• Probabilistic generative model learning latent distributions you can sample from • enables controlled generation, though image samples tend to be blurrier than a GAN's.
Autoencoder	`encoder = Sequential([Dense(128), Dense(64)])` `decoder = Sequential([Dense(128), Dense(input_dim)])`	• Unsupervised network trained to copy its input to its output through a bottleneck • used for dimensionality reduction, denoising, and feature learning.
U-Net	`encoder_output = Conv2D(...)(x)` `decoder_input = Concatenate()([upsampled, encoder_output])`	• CNN with a contracting encoder-decoder structure plus skip connections • outputs a per-pixel map at input resolution for image segmentation.
Recurrent Neural Network (RNN)	`model = Sequential([` `SimpleRNN(128, return_sequences=True),` `Dense(vocab_size)])`	• Sequential data processor with hidden state that carries temporal information • cannot parallelize across timesteps, and suffers vanishing gradients on long sequences.
Gated Recurrent Unit (GRU)	`model = Sequential([` `GRU(128, return_sequences=True),` `Dense(output_dim)])`	• Simplified LSTM with two gates (update, reset) and no separate cell state • fewer parameters and faster to compute, with comparable performance.
Bidirectional RNN	`model = Sequential([` `Bidirectional(LSTM(128)),` `Dense(output_dim)])`	• RNN processing sequences in both forward and backward directions • needs the whole sequence up front, so it cannot be used for streaming or causal generation.
AlexNet	`model = Sequential([` `Conv2D(96, 11, strides=4), MaxPooling2D(3, 2),` `Conv2D(256, 5), Flatten(), Dense(4096)])`	• 8-layer GPU-trained CNN that revolutionized ImageNet in 2012 • used ReLU and dropout to show learned features beat hand-designed ones.
VGG	`model = Sequential([` `Conv2D(64, 3), Conv2D(64, 3), MaxPooling2D(),` `Conv2D(128, 3), Conv2D(128, 3)])`	• Deep CNN using uniform 3×3 convolutions stacked in repeating blocks, 16-19 weight layers • two stacked 3×3 convs match one 5×5's receptive field with fewer parameters.
Inception (GoogLeNet)	`x = Concatenate()([` `Conv2D(64, 1)(x), Conv2D(128, 3)(x),` `Conv2D(32, 5)(x), MaxPooling2D(3)(x)])`	• Multi-scale CNN with four parallel branches of different kernel sizes • concatenates them so one block extracts features at several scales at once.

Table 2: Layer Types

Type	Example	Description
Dense (Fully Connected)	`Dense(128, activation='relu')`	• Connects every input neuron to every output neuron • parameters = inputs × outputs plus biases, which is why a dense head usually dominates a CNN's parameter count.

Table 3: Activation Functions

Function	Example	Description
ReLU (Rectified Linear Unit)	`Activation('relu')` `# f(x) = max(0, x)`	• Outputs the input if positive, zero otherwise • cheap to compute and non-saturating for positives, which greatly speeds up convergence.

Table 4: Optimization Algorithms

Optimizer	Example	Description
Adam (Adaptive Moment Estimation)	`Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999)`	• Tracks two running averages per weight: a smoothed gradient (momentum) and recent squared gradients (per-parameter step size) • both start at zero, so bias correction rescales them during early steps.