Neural Networks Core Cheat Sheet

Updated 2026-04-27

Next Topic: NVIDIA TensorRT for Inference Optimization Cheat Sheet

🧠Study flashcards on this topic112 cards · spaced repetition→

Neural networks are computational models inspired by biological neurons, consisting of interconnected layers of nodes (neurons) that learn patterns through backpropagation and gradient descent. They form the foundation of modern deep learning, enabling breakthroughs in computer vision, natural language processing, sequential data modeling, and generative AI. Key to success: network depth enables feature hierarchy, proper initialization and normalization prevent gradient issues, and the right architecture family — feedforward, recurrent, or attention-based — must match the data structure. Modern training practice combines adaptive optimizers like AdamW, mixed-precision arithmetic, and learning-rate schedules to train efficiently at scale.

What This Cheat Sheet Covers

This topic spans 14 focused tables and 125 indexed concepts, 112 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Network ArchitecturesTable 2: Layer TypesTable 3: Activation FunctionsTable 4: Loss FunctionsTable 5: Optimization AlgorithmsTable 6: Regularization TechniquesTable 7: Training MechanicsTable 8: Weight InitializationTable 9: Common Problems and SolutionsTable 10: HyperparametersTable 11: Evaluation MetricsTable 12: Pooling OperationsTable 13: Advanced TechniquesTable 14: Model Compression

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Network Architectures

The architecture you reach for is the single biggest decision in any deep learning project, because it has to match the shape of your data — feedforward nets for tabular inputs, CNNs for grids of pixels, RNNs and their gated variants for sequences, and transformers when long-range relationships matter most. Knowing what each family was built to solve, and the problem that motivated the next one (skip connections rescuing very deep nets, attention replacing recurrence), makes the rest of the cheat sheet click into place.

Type	Example	Description
Feedforward Neural Network (FNN)	`input → hidden1 → hidden2 → output`	• Information flows in one direction only from input to output • no cycles • simplest architecture for supervised learning.
Multilayer Perceptron (MLP)	`Dense(128, relu) → Dense(64, relu) → Dense(10, softmax)`	• Feedforward network with one or more fully connected hidden layers • standard for tabular data and classification.
Convolutional Neural Network (CNN)	`Conv2D → ReLU → MaxPool → Flatten → Dense`	• Specialized for spatial data (images) • uses convolutional filters to detect local patterns • dominant architecture for computer vision.
Recurrent Neural Network (RNN)	`h_t = tanh(W_h * h_{t-1} + W_x * x_t)`	• Maintains hidden state across time steps • processes sequential data (text, time series) • suffers from vanishing gradients on long sequences.
LSTM (Long Short-Term Memory)	`LSTM(units=128, return_sequences=True)`	• RNN variant with forget, input, and output gates controlling cell state • solves vanishing gradient for long-range dependencies.
GRU (Gated Recurrent Unit)	`GRU(units=128)`	• Simplified LSTM with reset and update gates but no separate cell state • fewer parameters than LSTM • competitive performance on many tasks.
Residual Network (ResNet)	`y = F(x) + x` (skip connection)	• Introduces skip connections adding input directly to layer output • solves vanishing gradient in very deep networks (100+ layers).
Autoencoder	`input → encoder → latent → decoder → output`	• Learns compressed latent representation in encoder, then reconstructs in decoder • used for dimensionality reduction, anomaly detection, generation.
Transformer	`MultiHeadAttention → FFN → LayerNorm (×N)`	• Uses self-attention to model all pairwise token relationships in parallel • no recurrence; dominant architecture for LLMs and modern NLP/vision.
Generative Adversarial Network (GAN)	Generator vs. Discriminator trained adversarially	• Generator creates fake samples; Discriminator distinguishes real from fake • trained via minimax game • produces high-fidelity synthetic data.
U-Net	Encoder–decoder with skip connections at each level	• Encoder–decoder CNN with symmetric skip connections preserving spatial detail • designed for biomedical image segmentation • widely used for dense prediction.
Perceptron	`output = step(w · x + b)`	• Single-layer binary classifier using step activation • historical foundation but limited to linearly separable problems.

Table 2: Layer Types

Architectures are assembled from a small vocabulary of layers, each with a distinct job — dense layers mix everything together, convolutions exploit spatial locality, recurrent layers carry state through time, and normalization and dropout layers exist purely to keep training stable and honest. Recognizing which layer does what lets you read any model definition like a sentence.

Type	Example	Description
Dense (Fully Connected)	`Dense(128, activation='relu')`	• Every neuron connects to every neuron in the previous layer • standard hidden layer for MLPs and classifier heads.

The rest of this section is part of the full cheat sheet

Table 3: Activation Functions

Without a non-linear activation between layers, a deep network collapses into a single linear transformation — these functions are what let it bend and fit complex patterns. ReLU and its smoother cousins dominate the hidden layers, while sigmoid and softmax usually sit at the output to shape predictions into probabilities; the rest of the table mostly chases the same goal of avoiding dead or vanishing gradients with a slightly better curve.

Function	Example	Description
ReLU (Rectified Linear Unit)	$f(x) = \max(0, x)$	• Returns x if positive, else 0 • most widely used • fast to compute • suffers from dying neurons when inputs consistently negative.

The rest of this section is part of the full cheat sheet

Table 4: Loss Functions

The loss function defines what "wrong" means, and training does nothing but shrink it — so picking the right one is how you tell the network what you actually care about. Cross-entropy variants handle classification, MSE and its relatives handle regression, and the more specialized losses here target specific pain points like class imbalance, embedding geometry, or pixel-level overlap in segmentation.

Function	Example	Description
Binary Cross-Entropy	$-\frac{1}{N}\sum [y\log(\hat{y}) + (1-y)\log(1-\hat{y})]$	• Standard loss for binary classification • heavily penalizes confident wrong predictions • used with sigmoid output.

The rest of this section is part of the full cheat sheet

Table 1: Network Architectures

Type	Example	Description
Feedforward Neural Network (FNN)	`input → hidden1 → hidden2 → output`	• Information flows in one direction only from input to output • no cycles • simplest architecture for supervised learning.
Multilayer Perceptron (MLP)	`Dense(128, relu) → Dense(64, relu) → Dense(10, softmax)`	• Feedforward network with one or more fully connected hidden layers • standard for tabular data and classification.
Convolutional Neural Network (CNN)	`Conv2D → ReLU → MaxPool → Flatten → Dense`	• Specialized for spatial data (images) • uses convolutional filters to detect local patterns • dominant architecture for computer vision.
Recurrent Neural Network (RNN)	`h_t = tanh(W_h * h_{t-1} + W_x * x_t)`	• Maintains hidden state across time steps • processes sequential data (text, time series) • suffers from vanishing gradients on long sequences.
LSTM (Long Short-Term Memory)	`LSTM(units=128, return_sequences=True)`	• RNN variant with forget, input, and output gates controlling cell state • solves vanishing gradient for long-range dependencies.
GRU (Gated Recurrent Unit)	`GRU(units=128)`	• Simplified LSTM with reset and update gates but no separate cell state • fewer parameters than LSTM • competitive performance on many tasks.
Residual Network (ResNet)	`y = F(x) + x` (skip connection)	• Introduces skip connections adding input directly to layer output • solves vanishing gradient in very deep networks (100+ layers).
Autoencoder	`input → encoder → latent → decoder → output`	• Learns compressed latent representation in encoder, then reconstructs in decoder • used for dimensionality reduction, anomaly detection, generation.
Transformer	`MultiHeadAttention → FFN → LayerNorm (×N)`	• Uses self-attention to model all pairwise token relationships in parallel • no recurrence; dominant architecture for LLMs and modern NLP/vision.
Generative Adversarial Network (GAN)	Generator vs. Discriminator trained adversarially	• Generator creates fake samples; Discriminator distinguishes real from fake • trained via minimax game • produces high-fidelity synthetic data.
U-Net	Encoder–decoder with skip connections at each level	• Encoder–decoder CNN with symmetric skip connections preserving spatial detail • designed for biomedical image segmentation • widely used for dense prediction.
Perceptron	`output = step(w · x + b)`	• Single-layer binary classifier using step activation • historical foundation but limited to linearly separable problems.

Table 2: Layer Types

Type	Example	Description
Dense (Fully Connected)	`Dense(128, activation='relu')`	• Every neuron connects to every neuron in the previous layer • standard hidden layer for MLPs and classifier heads.

Table 3: Activation Functions

Function	Example	Description
ReLU (Rectified Linear Unit)	$f(x) = \max(0, x)$	• Returns x if positive, else 0 • most widely used • fast to compute • suffers from dying neurons when inputs consistently negative.

Table 4: Loss Functions

Function	Example	Description
Binary Cross-Entropy	$-\frac{1}{N}\sum [y\log(\hat{y}) + (1-y)\log(1-\hat{y})]$	• Standard loss for binary classification • heavily penalizes confident wrong predictions • used with sigmoid output.