Uncertainty quantification and prediction calibration form the foundation of trustworthy machine learning — the difference between a model that predicts "90% confident" and one where 90% confidence actually means 90% accuracy. These techniques span Bayesian approximations (Monte Carlo dropout, variational inference, Laplace), ensemble-based approaches (deep ensembles, SWAG), post-hoc calibration methods (temperature scaling, Platt scaling), conformal prediction for distribution-free guarantees, and metrics like ECE and Brier score that quantify calibration quality. Two fundamental types of uncertainty drive this field: epistemic uncertainty from model ignorance (reducible with more data or better architectures) and aleatoric uncertainty from irreducible data noise. Whether deploying safety-critical medical AI, building production recommenders that know when to abstain, or quantifying prediction intervals for regression, these methods bridge the gap between raw model outputs and interpretable, actionable confidence scores — a crucial step toward AI systems humans can trust.
What This Cheat Sheet Covers
This topic spans 19 focused tables and 99 indexed concepts. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
Table 1: Core Uncertainty Types and Concepts
Before any method makes sense, you need the vocabulary — and the single most important distinction here is epistemic versus aleatoric uncertainty, because it tells you whether more data will help. Epistemic uncertainty comes from the model not knowing enough and shrinks with better data or architectures; aleatoric uncertainty is irreducible noise baked into the observations themselves. The remaining rows — heteroscedastic versus homoscedastic noise, total predictive uncertainty, and OOD detection — are all consequences of how those two sources combine and surface at prediction time.
| Type | Example | Description |
|---|---|---|
Monte Carlo dropout with 100 forward passes: variance in predictions reflects epistemic uncertainty | • Uncertainty from model ignorance — reducible by collecting more training data, using better architectures, or longer training • captured by variance in model parameters or predictions across different plausible models | |
Predicting pixel color from a blurry image: inherent noise in image means high aleatoric uncertainty | • Uncertainty from irreducible data noise — cannot be reduced even with infinite data • intrinsic randomness in observations (sensor noise, label ambiguity, stochasticity in the process itself). | |
Neural net predicts mean \mu(x) and variance \sigma^2(x) as separate outputs | Aleatoric uncertainty that varies across input space — model learns input-dependent noise levels (e.g., low confidence on edge cases, high on common patterns). |