Model Monitoring and Drift Detection Cheat Sheet

Updated 2026-05-18

Next Topic: Model Pruning and Neural Network Compression Cheat Sheet

Model monitoring is the continuous process of tracking machine learning models in production to ensure they maintain performance as real-world data evolves. Drift detection identifies when the statistical properties of data or relationships between inputs and outputs change over time, signaling potential model degradation. Understanding the types of drift, detection methods, and monitoring strategies is essential for maintaining reliable ML systems—catching performance issues before they impact users requires both statistical rigor and operational infrastructure.

What This Cheat Sheet Covers

This topic spans 12 focused tables and 71 indexed concepts. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Types of Drift in ML ProductionTable 2: Statistical Distance Metrics for Drift DetectionTable 3: Hypothesis Testing and Sequential Detection MethodsTable 4: Monitoring Frameworks and Open-Source ToolsTable 5: Cloud Platform Monitoring SolutionsTable 6: Deployment Patterns for Safe Model UpdatesTable 7: Performance Estimation Without Ground TruthTable 8: Alerting Strategies and Retraining TriggersTable 9: Multivariate and Advanced Drift DetectionTable 10: Data Quality Monitoring for MLTable 11: Specialized Monitoring for Advanced Use CasesTable 12: Operational Best Practices and Infrastructure

Table 1: Types of Drift in ML Production

Drift in machine learning occurs when the data or relationships your model relies on change over time, degrading predictions. Understanding these distinct drift types helps you pinpoint root causes and apply targeted fixes rather than blindly retraining.

Type	Example	Description
Data Drift (Covariate Shift)	$P(X)$ changes but $P(Y\|X)$ remains constant; image resolution changes in production but object labels stay the same	Input distribution shifts while the relationship between features and target stays intact • Most common drift type in production • Detected via distribution distance metrics on features
Concept Drift	$P(Y\|X)$ changes; customer preferences shift, spam evolves, fraud patterns change	Relationship between inputs and outputs evolves • Requires relabeling or model retraining • Hardest drift to detect without ground truth
Label Drift (Prior Probability Shift)	$P(Y)$ changes but $P(X\|Y)$ stays constant; class imbalance shifts from 50/50 to 80/20 in production	Target variable distribution changes independent of feature values • Often caused by seasonality or sampling bias • Impacts model calibration and decision thresholds
Prediction Drift	Model outputs shift unexpectedly; predicted probabilities cluster at extremes or flatten	Model's output distribution changes over time • Can signal data drift, concept drift, or model degradation • Early warning signal when ground truth is delayed
Feature Drift	Individual feature values shift; average income increases, age distribution skews younger	Specific input features change in distribution or scale • Subset of data drift focused on per-feature analysis • High-importance features drifting cause more impact

Table 1: Types of Drift in ML Production

Type	Example	Description
Data Drift (Covariate Shift)	$P(X)$ changes but $P(Y\|X)$ remains constant; image resolution changes in production but object labels stay the same	Input distribution shifts while the relationship between features and target stays intact • Most common drift type in production • Detected via distribution distance metrics on features
Concept Drift	$P(Y\|X)$ changes; customer preferences shift, spam evolves, fraud patterns change	Relationship between inputs and outputs evolves • Requires relabeling or model retraining • Hardest drift to detect without ground truth
Label Drift (Prior Probability Shift)	$P(Y)$ changes but $P(X\|Y)$ stays constant; class imbalance shifts from 50/50 to 80/20 in production	Target variable distribution changes independent of feature values • Often caused by seasonality or sampling bias • Impacts model calibration and decision thresholds
Prediction Drift	Model outputs shift unexpectedly; predicted probabilities cluster at extremes or flatten	Model's output distribution changes over time • Can signal data drift, concept drift, or model degradation • Early warning signal when ground truth is delayed
Feature Drift	Individual feature values shift; average income increases, age distribution skews younger	Specific input features change in distribution or scale • Subset of data drift focused on per-feature analysis • High-importance features drifting cause more impact