Machine Learning System Design Cheat Sheet

Updated 2026-05-18

Next Topic: Mixture of Experts (MoE) Architecture Cheat Sheet

🧠Study flashcards on this topic122 cards · spaced repetition→

Machine learning system design is the architectural discipline of building end-to-end ML systems that operate reliably at scale in production. Unlike traditional software systems, ML systems must handle probabilistic outputs, continuous data evolution, and the unique challenge of serving predictions while simultaneously learning from new data. Modern ML system design integrates data pipelines, training infrastructure, model serving, experimentation frameworks, and monitoring systems into a cohesive architecture. The most critical distinction: ML systems degrade silently — without proper monitoring and retraining triggers, model performance erodes invisibly as the world changes beneath them.

What This Cheat Sheet Covers

This topic spans 15 focused tables and 149 indexed concepts, 122 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Core ML System Design PatternsTable 2: Feature Engineering InfrastructureTable 3: Model Training Pipeline ArchitectureTable 4: Model Registry and VersioningTable 5: Model Serving InfrastructureTable 6: Serving Performance OptimizationTable 7: Experimentation and Online EvaluationTable 8: Model Monitoring and ObservabilityTable 9: Drift Detection TechniquesTable 10: Retraining Strategies and TriggersTable 11: Data Pipeline ArchitectureTable 12: Distributed Training StrategiesTable 13: Inference Patterns and Trade-offsTable 14: ML System Security and PrivacyTable 15: Cost Optimization Strategies

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Core ML System Design Patterns

Production ML systems follow repeatable architectural patterns that balance prediction accuracy, latency, cost, and maintainability. These patterns represent proven approaches for deploying models at scale.

Pattern	Example	Description
Batch Prediction Pipeline	`predictions = model.predict(daily_data)` `store_to_cache(predictions)`	• Precomputes predictions for all inputs on a schedule (hourly/daily) • serves results from cache for low-latency lookups at the cost of staleness
Real-Time Inference Service	`@app.post("/predict")` `return model.predict(request.features)`	• Computes predictions on-demand per request • sub-100ms latency requirement drives optimization choices like model size and caching
Online Learning System	`model.partial_fit(new_batch)` `if drift_detected: retrain()`	• Updates model continuously with streaming data • trades training stability for adaptiveness to distribution shifts in real-time
Feature Store Architecture	`features = store.get_online(user_id)` `offline = store.get_historical(timestamp)`	• Centralized feature computation and serving layer • ensures training-serving consistency by using identical feature logic in both paths
Model Registry Pattern	`registry.log_model(model, metrics)` `prod_model = registry.load("v2.3")`	• Version control for trained models with lineage tracking • enables rollbacks, A/B testing, and audit trails of what ran when
Shadow Deployment	`old_pred = model_v1.predict(x)` `new_pred = model_v2.predict(x)` `log_both(); serve(old_pred)`	• Runs new model in parallel with production model without affecting users • validates performance on live traffic before cutover
Canary Release	`if user_id % 100 < 5:` `return new_model.predict(x)`	• Gradually shifts traffic to new model (5% → 25% → 100%) • limits blast radius of model regressions while collecting real-world metrics
A/B Testing Framework	`model = assign_variant(user_id)` `log_prediction_and_outcome()`	• Compares models by randomly assigning users to variants • measures causal impact on business metrics rather than just accuracy
Multi-Armed Bandit	`probs = thompson_sampling(rewards)` `model = np.random.choice(models, p=probs)`	• Dynamically allocates traffic based on performance • optimizes while learning unlike fixed A/B tests that wait for significance
Hybrid Batch-Real-Time	`cached = redis.get(key)` `if cached: return cached` `return model.predict(input)`	• Precomputes 90% of predictions in batch • falls back to real-time inference on cache miss for cost-effective low latency

Table 2: Feature Engineering Infrastructure

Feature engineering at scale requires infrastructure that computes, stores, and serves features consistently across training and serving environments while handling both batch and real-time computation patterns.

Component	Example	Description
Feature Store	`store.register_feature_view("user_stats")` `features = store.get_online(user_id)`	• Centralized repository for feature definitions, storage, and serving • point-in-time correctness prevents data leakage during training

The rest of this section is part of the full cheat sheet

Table 3: Model Training Pipeline Architecture

Large-scale model training requires orchestrating data pipelines, compute resources, experiment tracking, and model versioning into automated workflows that support both batch training and continuous retraining.

Component	Example	Description
ML Pipeline Orchestration	`@pipeline(schedule="daily")` `def train(): ingest() → preprocess() → train()`	• Coordinates multi-step workflows (data prep, training, validation, deployment) • tools like Kubeflow and Airflow handle dependencies and retries

The rest of this section is part of the full cheat sheet

Table 4: Model Registry and Versioning

Model registries provide centralized management of trained models with version control, lineage tracking, and lifecycle management from development through production deployment.

Concept	Example	Description
Model Registration	`mlflow.register_model(model_uri, "forecaster")` `version = registry.create_version("v2.1")`	• Catalogs trained models with unique identifiers • links models to training runs with full hyperparameter and dataset provenance

The rest of this section is part of the full cheat sheet

Table 1: Core ML System Design Patterns

Pattern	Example	Description
Batch Prediction Pipeline	`predictions = model.predict(daily_data)` `store_to_cache(predictions)`	• Precomputes predictions for all inputs on a schedule (hourly/daily) • serves results from cache for low-latency lookups at the cost of staleness
Real-Time Inference Service	`@app.post("/predict")` `return model.predict(request.features)`	• Computes predictions on-demand per request • sub-100ms latency requirement drives optimization choices like model size and caching
Online Learning System	`model.partial_fit(new_batch)` `if drift_detected: retrain()`	• Updates model continuously with streaming data • trades training stability for adaptiveness to distribution shifts in real-time
Feature Store Architecture	`features = store.get_online(user_id)` `offline = store.get_historical(timestamp)`	• Centralized feature computation and serving layer • ensures training-serving consistency by using identical feature logic in both paths
Model Registry Pattern	`registry.log_model(model, metrics)` `prod_model = registry.load("v2.3")`	• Version control for trained models with lineage tracking • enables rollbacks, A/B testing, and audit trails of what ran when
Shadow Deployment	`old_pred = model_v1.predict(x)` `new_pred = model_v2.predict(x)` `log_both(); serve(old_pred)`	• Runs new model in parallel with production model without affecting users • validates performance on live traffic before cutover
Canary Release	`if user_id % 100 < 5:` `return new_model.predict(x)`	• Gradually shifts traffic to new model (5% → 25% → 100%) • limits blast radius of model regressions while collecting real-world metrics
A/B Testing Framework	`model = assign_variant(user_id)` `log_prediction_and_outcome()`	• Compares models by randomly assigning users to variants • measures causal impact on business metrics rather than just accuracy
Multi-Armed Bandit	`probs = thompson_sampling(rewards)` `model = np.random.choice(models, p=probs)`	• Dynamically allocates traffic based on performance • optimizes while learning unlike fixed A/B tests that wait for significance
Hybrid Batch-Real-Time	`cached = redis.get(key)` `if cached: return cached` `return model.predict(input)`	• Precomputes 90% of predictions in batch • falls back to real-time inference on cache miss for cost-effective low latency

Table 2: Feature Engineering Infrastructure

Component	Example	Description
Feature Store	`store.register_feature_view("user_stats")` `features = store.get_online(user_id)`	• Centralized repository for feature definitions, storage, and serving • point-in-time correctness prevents data leakage during training

Table 3: Model Training Pipeline Architecture

Component	Example	Description
ML Pipeline Orchestration	`@pipeline(schedule="daily")` `def train(): ingest() → preprocess() → train()`	• Coordinates multi-step workflows (data prep, training, validation, deployment) • tools like Kubeflow and Airflow handle dependencies and retries

Table 4: Model Registry and Versioning

Model registries provide centralized management of trained models with version control, lineage tracking, and lifecycle management from development through production deployment.

Concept	Example	Description
Model Registration	`mlflow.register_model(model_uri, "forecaster")` `version = registry.create_version("v2.1")`	• Catalogs trained models with unique identifiers • links models to training runs with full hyperparameter and dataset provenance