Change Data Capture (CDC) is a design pattern that tracks and streams every change (insert, update, delete) made to a dataset in near real-time. Originally developed for database replication, CDC has evolved into the backbone of modern event-driven architectures, enabling real-time analytics, microservices synchronization, and data lake ingestion without impacting source system performance. Unlike batch ETL which periodically polls tables, CDC reads transaction logs—the append-only journal every ACID database maintains—and converts those low-level log events into structured change streams. This log-based approach delivers sub-second latency at a fraction of the resource cost, making CDC the de facto standard for keeping analytical and operational systems in sync across OLTP databases, distributed systems, and open-format lakehouses.
What This Cheat Sheet Covers
This topic spans 21 focused tables and 127 indexed concepts, 123 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
A jump-to index of every table row in this cheat sheet.
An interactive map of every table and concept in this topic.
Table 1: CDC Implementation Approaches
The six fundamental approaches differ in where changes are captured (log, trigger, or query), who initiates the transfer (push vs. pull), and what trade-offs they make between latency, source load, and operational complexity.
| Approach | Example | Description | |
|---|---|---|---|
PostgreSQL WAL → DebeziumMySQL binlog → Kafka Connect | • Reads database transaction logs (WAL, binlog, redo logs) to capture every change without querying tables • lowest latency (sub-second) and zero source impact but requires log access permissions. | ||
Initial full table dump→ switch to log-based CDC | • Combines full table snapshot for historical data with log-based capture for ongoing changes • standard pattern for backfilling analytical systems while maintaining real-time sync. | ||
CREATE TRIGGER on_updateINSERT INTO cdc_table | • Database triggers write change records to a shadow table on every INSERT/UPDATE/DELETE • simple to set up but adds write overhead to every transaction and may miss DDL changes. | ||
SELECT * WHERE updated_at >MAX(last_sync_time) | • Periodically polls tables filtering by timestamp or sequence column • highest latency (minutes) and cannot detect hard deletes, but works when log access is unavailable. | ||
Database publishes changes → CDC framework | • Source database actively pushes change events to downstream systems • database-driven but requires native CDC capabilities like SQL Server CDC or MongoDB Change Streams. | ||
CDC framework polls → database responds | • External tool periodically reads transaction logs or change tables from the database • most common for heterogeneous environments where source systems lack native CDC. |