Data Validation and Quality in Data Science Cheat Sheet

Updated 2026-05-28

Next Topic: Data Visualization Cheat Sheet

🧠Study flashcards on this topic141 cards · spaced repetition→

Data validation and quality management form the critical foundation of reliable data science workflows, ensuring that models train on trustworthy inputs and produce dependable predictions. In 2026, the shift from reactive quality checks to proactive data observability has transformed validation from a one-time ingestion step into a continuous process spanning feature engineering, model training, and production monitoring. This cheat sheet covers validation techniques from foundational schema checks through advanced statistical drift detection, emphasizing that quality gates at every pipeline stage prevent downstream model failures and maintain trust in AI systems.

What This Cheat Sheet Covers

This topic spans 26 focused tables and 190 indexed concepts, 141 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Data Quality DimensionsTable 2: Schema Validation TechniquesTable 3: Type and Range ValidationTable 4: Null and Uniqueness ConstraintsTable 5: Python Validation FrameworksTable 6: Expectation-Based TestingTable 7: DataFrame Contracts (Pandera)Table 8: Data Profiling ApproachesTable 9: Statistical Testing MethodsTable 10: Drift Detection TechniquesTable 11: Anomaly Detection MethodsTable 12: Quality Gates and CI/CD IntegrationTable 13: Quality Monitoring and ObservabilityTable 14: pytest Integration for DataTable 15: dbt Data TestingTable 16: Constraint-Based ValidationTable 17: Data Profiling with ydata-profilingTable 18: Validation Decorators and AnnotationsTable 19: Real-Time and Streaming ValidationTable 20: Feature Store ValidationTable 21: Data Versioning and ComparisonTable 22: Statistical Hypothesis TestingTable 23: Documentation and Metadata ManagementTable 24: DataOps and Continuous ValidationTable 25: Advanced Validation PatternsTable 26: Quality Monitoring Platforms (2026)

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Data Quality Dimensions

Core dimensions used to assess fitness-for-purpose of data assets. Every serious data quality program measures along these axes; neglecting any single dimension typically produces downstream errors that are hard to trace back to source.

Dimension	Example	Description
Completeness	`completeness = 1 - df.isnull().mean()`	• Proportion of required data present • critical for avoiding sampling bias in ML models
Accuracy	`correct_ratio = (df['state'].isin(valid_states)).mean()`	• Degree to which data correctly represents real-world entities • measured by comparing against authoritative sources
Validity	`email_valid = df['email'].str.match(r'^[^@]+@[^@]+\.[^@]+$')`	• Conformance to defined formats, types, and business rules • ensures data adheres to domain constraints
Consistency	`assert (df['end_date'] >= df['start_date']).all()`	• Agreement across multiple records or systems • ensures referential integrity in related datasets
Uniqueness	`duplicates = df.duplicated(subset=['id']).sum()`	• Absence of unwanted duplicate records • crucial for preventing double-counting in aggregations
Timeliness	`lag = datetime.now() - df['last_updated'].max()`	• How current and up-to-date data is • measured via data freshness and latency from source to destination

Table 2: Schema Validation Techniques

Approaches for enforcing structural requirements and data types. Choosing the right schema format depends on your transport layer — JSON for REST APIs, Avro for Kafka streams, Protobuf for gRPC services.

Technique	Example	Description
Type Checking	`from pydantic import BaseModel` `class User(BaseModel):` `age: int # Type validated automatically`	• Enforce data types at runtime using Python type hints and validation libraries • Pydantic v2 offers 17x speed improvements via Rust core

The rest of this section is part of the full cheat sheet

Table 3: Type and Range Validation

Constraint checks for individual field values. These are the most basic and fastest checks to run — apply them at the earliest possible pipeline stage to avoid propagating bad values.

Technique	Example	Description
Range Constraints	`pa.Column(float, pa.Check.in_range(0, 100))`	• Validate numeric values fall within acceptable bounds • catches out-of-domain values before model training

The rest of this section is part of the full cheat sheet

Table 4: Null and Uniqueness Constraints

Checks for completeness and record-level integrity. Null propagation and unexpected duplicates are among the most common causes of silent data corruption in production pipelines.

Technique	Example	Description
Null/Not-Null Checks	`pa.Column(str, nullable=False) # Disallow nulls`	• Enforce required fields • prevents missing value propagation through downstream transformations

The rest of this section is part of the full cheat sheet

Table 1: Data Quality Dimensions

Dimension	Example	Description
Completeness	`completeness = 1 - df.isnull().mean()`	• Proportion of required data present • critical for avoiding sampling bias in ML models
Accuracy	`correct_ratio = (df['state'].isin(valid_states)).mean()`	• Degree to which data correctly represents real-world entities • measured by comparing against authoritative sources
Validity	`email_valid = df['email'].str.match(r'^[^@]+@[^@]+\.[^@]+$')`	• Conformance to defined formats, types, and business rules • ensures data adheres to domain constraints
Consistency	`assert (df['end_date'] >= df['start_date']).all()`	• Agreement across multiple records or systems • ensures referential integrity in related datasets
Uniqueness	`duplicates = df.duplicated(subset=['id']).sum()`	• Absence of unwanted duplicate records • crucial for preventing double-counting in aggregations
Timeliness	`lag = datetime.now() - df['last_updated'].max()`	• How current and up-to-date data is • measured via data freshness and latency from source to destination

Table 2: Schema Validation Techniques

Technique	Example	Description
Type Checking	`from pydantic import BaseModel` `class User(BaseModel):` `age: int # Type validated automatically`	• Enforce data types at runtime using Python type hints and validation libraries • Pydantic v2 offers 17x speed improvements via Rust core

Table 3: Type and Range Validation

Constraint checks for individual field values. These are the most basic and fastest checks to run — apply them at the earliest possible pipeline stage to avoid propagating bad values.

Technique	Example	Description
Range Constraints	`pa.Column(float, pa.Check.in_range(0, 100))`	• Validate numeric values fall within acceptable bounds • catches out-of-domain values before model training

Table 4: Null and Uniqueness Constraints

Checks for completeness and record-level integrity. Null propagation and unexpected duplicates are among the most common causes of silent data corruption in production pipelines.

Technique	Example	Description
Null/Not-Null Checks	`pa.Column(str, nullable=False) # Disallow nulls`	• Enforce required fields • prevents missing value propagation through downstream transformations