Mathematics for machine learning is the formal language underlying every algorithm that learns from data. It spans linear algebra, multivariate calculus, probability theory, information theory, and optimization — topics that appear in every paper, every framework, and every model architecture. Without this foundation it is possible to use ML tools as black boxes, but understanding it is what separates practitioners who can debug, design, and innovate from those who can only configure. The crucial mental model: most ML is simply optimization of a scalar loss function over a high-dimensional parameter space, and every concept here — from eigenvectors to KL divergence — is a tool for understanding or improving that optimization.
What This Cheat Sheet Covers
This topic spans 18 focused tables and 133 indexed concepts, 133 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
A jump-to index of every table row in this cheat sheet.
An interactive map of every table and concept in this topic.
Table 1: Vectors, Matrices, and Tensors
Linear algebra gives ML its fundamental data structures. A scalar is a single number, a vector is an ordered array, a matrix is a 2-D grid, and a tensor generalizes to arbitrary dimensions — these are the objects that represent data, parameters, and transformations throughout every ML pipeline.
| Concept | Example | Description | |
|---|---|---|---|
$\mathbf{x} = [x_1, x_2, \ldots, x_n]^\top \in \mathbb{R}^n$ | • Ordered list of $n$ real numbers • represents a data point, a weight sequence, or an embedding | ||
$A \in \mathbb{R}^{m \times n}$, element $A_{ij}$ | • 2-D array of numbers • a dataset of $m$ examples with $n$ features forms an $m \times n$ matrix | ||
$\mathcal{T} \in \mathbb{R}^{H \times W \times C}$ (e.g., RGB image) | • Generalization of matrices to $k$ dimensions • the native data type in deep learning frameworks | ||
$\mathbf{u} \cdot \mathbf{v} = \sum_i u_i v_i = \lVert \mathbf{u} \rVert \lVert \mathbf{v} \rVert \cos\theta$ | • Produces a scalar • cosine similarity uses the normalized form to measure directional similarity between vectors | ||
$(AB){ij} = \sum_k A{ik} B_{kj}$, $A \in \mathbb{R}^{m\times k}$, $B \in \mathbb{R}^{k\times n}$ | • Core neural-network operation • each layer applies a linear transformation $A\mathbf{x} + \mathbf{b}$ via matrix-vector product | ||
$(A^\top){ij} = A{ji}$ | • Flips rows and columns • used constantly to align dimensions and in the relation $\langle \mathbf{u}, A\mathbf{v} \rangle = \langle A^\top \mathbf{u}, \mathbf{v} \rangle$. | ||
$(A \odot B){ij} = A{ij} B_{ij}$ | • Entry-wise multiplication • used in gating mechanisms (LSTM, attention masks). | ||
$\mathbf{u} \mathbf{v}^\top \in \mathbb{R}^{m \times n}$, $(\mathbf{u}\mathbf{v}^\top)_{ij} = u_i v_j$ | • Produces a rank-1 matrix • building block for low-rank approximations | ||
$I\mathbf{x} = \mathbf{x}$, diagonal of 1s | • Multiplicative identity • $AA^{-1} = I$ defines invertibility | ||
$A = A^\top$, e.g., covariance matrix $\Sigma$ | Symmetric matrices arise naturally as covariance or kernel matrices and have real eigenvalues (spectral theorem). |