Model Monitoring and Drift Detection Cheat Sheet

Updated 2026-05-18

Next Topic: Model Pruning and Neural Network Compression Cheat Sheet

🧠Study flashcards on this topic70 cards · spaced repetition→

Model monitoring is the continuous process of tracking machine learning models in production to ensure they maintain performance as real-world data evolves. Drift detection identifies when the statistical properties of data or relationships between inputs and outputs change over time, signaling potential model degradation. Understanding the types of drift, detection methods, and monitoring strategies is essential for maintaining reliable ML systems—catching performance issues before they impact users requires both statistical rigor and operational infrastructure.

What This Cheat Sheet Covers

This topic spans 12 focused tables and 71 indexed concepts, 70 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Types of Drift in ML ProductionTable 2: Statistical Distance Metrics for Drift DetectionTable 3: Hypothesis Testing and Sequential Detection MethodsTable 4: Monitoring Frameworks and Open-Source ToolsTable 5: Cloud Platform Monitoring SolutionsTable 6: Deployment Patterns for Safe Model UpdatesTable 7: Performance Estimation Without Ground TruthTable 8: Alerting Strategies and Retraining TriggersTable 9: Multivariate and Advanced Drift DetectionTable 10: Data Quality Monitoring for MLTable 11: Specialized Monitoring for Advanced Use CasesTable 12: Operational Best Practices and Infrastructure

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Types of Drift in ML Production

Drift in machine learning occurs when the data or relationships your model relies on change over time, degrading predictions. Understanding these distinct drift types helps you pinpoint root causes and apply targeted fixes rather than blindly retraining.

Type	Example	Description
Data Drift (Covariate Shift)	$P(X)$ changes but $P(Y\|X)$ remains constant; image resolution changes in production but object labels stay the same	• Input distribution shifts while the relationship between features and target stays intact • Most common drift type in production • Detected via distribution distance metrics on features
Concept Drift	$P(Y\|X)$ changes; customer preferences shift, spam evolves, fraud patterns change	• Relationship between inputs and outputs evolves • Requires relabeling or model retraining • Hardest drift to detect without ground truth
Label Drift (Prior Probability Shift)	$P(Y)$ changes but $P(X\|Y)$ stays constant; class imbalance shifts from 50/50 to 80/20 in production	• Target variable distribution changes independent of feature values • Often caused by seasonality or sampling bias • Impacts model calibration and decision thresholds
Prediction Drift	Model outputs shift unexpectedly; predicted probabilities cluster at extremes or flatten	• Model's output distribution changes over time • Can signal data drift, concept drift, or model degradation • Early warning signal when ground truth is delayed
Feature Drift	Individual feature values shift; average income increases, age distribution skews younger	• Specific input features change in distribution or scale • Subset of data drift focused on per-feature analysis • High-importance features drifting cause more impact
Sudden Drift	Model accuracy drops instantly after system upgrade or policy change	• Abrupt change in data or relationships • Easiest drift pattern to detect • Often caused by external events or upstream system changes
Gradual Drift	Slow decline in model accuracy over weeks or months as user behavior evolves	• Smooth, continuous change over extended period • Harder to detect than sudden drift • Requires trend-based detection with historical baselines
Incremental Drift	Step-wise changes in model performance; accuracy drops slightly every few weeks	• Sequential small shifts that accumulate over time • Middle ground between sudden and gradual • Detected via change-point algorithms
Recurring (Seasonal) Drift	Patterns cycle with time; holiday shopping behavior, quarterly business cycles	• Periodic patterns that repeat at regular intervals • Requires time-aware detection • Models can be adapted to expect seasonal shifts

Table 2: Statistical Distance Metrics for Drift Detection

Distance metrics quantify how much two probability distributions differ, forming the foundation for univariate drift detection. Choosing the right metric depends on your data type, sensitivity requirements, and computational constraints.

Metric	Example	Description
Population Stability Index (PSI)	$PSI = \sum (p_{prod} - p_{train}) \times \ln(p_{prod} / p_{train})$; PSI < 0.1 = no drift, PSI > 0.25 = significant drift	• Measures distribution shift between training and production data using binned frequencies • PSI thresholds (0.1, 0.25) are industry standards • Sensitive to bin selection and small sample sizes

The rest of this section is part of the full cheat sheet

Table 3: Hypothesis Testing and Sequential Detection Methods

Hypothesis testing methods monitor error rates or performance metrics over time, triggering drift alerts when statistical thresholds are crossed. These methods are particularly effective for detecting sudden and gradual concept drift in classification tasks.

Method	Example	Description
Drift Detection Method (DDM)	Monitors classification error rate $p$ and standard deviation $s$; drift detected when $p + s$ exceeds warning or drift threshold	• Tracks error rate distribution using confidence intervals • Assumes errors follow binomial distribution • Effective for gradual and sudden drift but struggles with very slow changes

The rest of this section is part of the full cheat sheet

Table 4: Monitoring Frameworks and Open-Source Tools

These libraries and frameworks provide prebuilt drift detection algorithms, performance estimation, and visualization capabilities. They accelerate monitoring implementation and integrate into ML pipelines.

Tool	Example	Description
Evidently AI	`from evidently.metrics import DataDriftTable` `report.run(reference_data=train, current_data=prod)`	• Open-source ML observability framework with 100+ metrics • Supports drift detection, performance estimation, and data quality checks • Generates interactive HTML reports and JSON profiles

The rest of this section is part of the full cheat sheet

Table 1: Types of Drift in ML Production

Type	Example	Description
Data Drift (Covariate Shift)	$P(X)$ changes but $P(Y\|X)$ remains constant; image resolution changes in production but object labels stay the same	• Input distribution shifts while the relationship between features and target stays intact • Most common drift type in production • Detected via distribution distance metrics on features
Concept Drift	$P(Y\|X)$ changes; customer preferences shift, spam evolves, fraud patterns change	• Relationship between inputs and outputs evolves • Requires relabeling or model retraining • Hardest drift to detect without ground truth
Label Drift (Prior Probability Shift)	$P(Y)$ changes but $P(X\|Y)$ stays constant; class imbalance shifts from 50/50 to 80/20 in production	• Target variable distribution changes independent of feature values • Often caused by seasonality or sampling bias • Impacts model calibration and decision thresholds
Prediction Drift	Model outputs shift unexpectedly; predicted probabilities cluster at extremes or flatten	• Model's output distribution changes over time • Can signal data drift, concept drift, or model degradation • Early warning signal when ground truth is delayed
Feature Drift	Individual feature values shift; average income increases, age distribution skews younger	• Specific input features change in distribution or scale • Subset of data drift focused on per-feature analysis • High-importance features drifting cause more impact
Sudden Drift	Model accuracy drops instantly after system upgrade or policy change	• Abrupt change in data or relationships • Easiest drift pattern to detect • Often caused by external events or upstream system changes
Gradual Drift	Slow decline in model accuracy over weeks or months as user behavior evolves	• Smooth, continuous change over extended period • Harder to detect than sudden drift • Requires trend-based detection with historical baselines
Incremental Drift	Step-wise changes in model performance; accuracy drops slightly every few weeks	• Sequential small shifts that accumulate over time • Middle ground between sudden and gradual • Detected via change-point algorithms
Recurring (Seasonal) Drift	Patterns cycle with time; holiday shopping behavior, quarterly business cycles	• Periodic patterns that repeat at regular intervals • Requires time-aware detection • Models can be adapted to expect seasonal shifts

Table 2: Statistical Distance Metrics for Drift Detection

Metric	Example	Description
Population Stability Index (PSI)	$PSI = \sum (p_{prod} - p_{train}) \times \ln(p_{prod} / p_{train})$; PSI < 0.1 = no drift, PSI > 0.25 = significant drift	• Measures distribution shift between training and production data using binned frequencies • PSI thresholds (0.1, 0.25) are industry standards • Sensitive to bin selection and small sample sizes

Table 3: Hypothesis Testing and Sequential Detection Methods

Method	Example	Description
Drift Detection Method (DDM)	Monitors classification error rate $p$ and standard deviation $s$; drift detected when $p + s$ exceeds warning or drift threshold	• Tracks error rate distribution using confidence intervals • Assumes errors follow binomial distribution • Effective for gradual and sudden drift but struggles with very slow changes

Table 4: Monitoring Frameworks and Open-Source Tools

Tool	Example	Description
Evidently AI	`from evidently.metrics import DataDriftTable` `report.run(reference_data=train, current_data=prod)`	• Open-source ML observability framework with 100+ metrics • Supports drift detection, performance estimation, and data quality checks • Generates interactive HTML reports and JSON profiles