New: Roadmaps ordered paths through our cheat sheets and flashcards, so you always know what to study next.
Explore themSee what's new on GitHubFrom your first SQL query to a Spark and Kafka pipeline that runs itself.
A 16-step learning path. Follow it in order, or jump to what you need.
This path is for developers, analysts, or DBAs who want to build and run the pipelines that move data at scale, not just query a warehouse someone else already built. Plan on about 7 to 11 weeks at a few hours a week, moving from Python and a quick SQL refresher through dimensional modeling, distributed processing with Spark, orchestration, and streaming. It does not cover statistics, experimentation, or machine learning, that ground belongs to the Data Scientist path, and it treats SQL as a launchpad rather than teaching relational theory or PostgreSQL administration in depth, which the SQL & Relational Databases path already owns. By the end you can model a warehouse schema that survives change, orchestrate a Spark pipeline on a schedule, and stream real-time data through Kafka into a lakehouse.
No prior experience needed. Start from zero.
Open this first and write working Python today: it's the language every later step here runs on, from Spark jobs to Airflow DAGs.
Put the code you just wrote in step 1 under real version control, the same habit every pipeline and dbt project in this path expects you to already have.
Pull real rows out of a database today; this path treats SQL as your entry ticket, the query language step 4's warehouse and step 10's dbt models both assume you already speak.
You can write Python, track it in Git, and pull real rows out of a database with SQL. Next up: turning those rows into a warehouse worth querying.
Finish this section to unlock.
+100 XP
See why a warehouse is built differently from the transactional database SQL just showed you: star schemas, fact tables, and the OLAP thinking every later modeling step here builds on.
Go deeper on the star schema step 4 sketched: the dimensional modeling method most production warehouses actually follow, and the one dbt models in step 10 are built to match.
Move data into the warehouse you just modeled: extraction, transformation, and loading patterns that turn steps 4 and 5's schema from a diagram into something rows actually land in.
This is where a single warehouse stops being enough: the distributed-systems thinking behind petabyte-scale data, expect the mental model to feel foreign before it clicks.
You can design a dimensional model and name exactly where a single warehouse hits its limit, the moment distributed thinking stops being optional (a few minutes of due flashcards keeps chapter 1's SQL sharp while you're here). Next up: putting Spark to work on data too big for one machine.
Finish this section to unlock.
+100 XP
Put big data's distributed thinking from step 7 into your own hands: PySpark is how you actually process data too large for the ETL patterns from step 6 to handle alone.
Schedule the Spark and ETL jobs from steps 6 and 8 to run themselves: DAGs, retries, and monitoring are what turn a script you run by hand into a pipeline a team can trust.
Turn the raw loads Airflow now schedules into tested, documented models: dbt applies the dimensional design from step 5 as version-controlled SQL instead of a one-off script.
Stop waiting for tomorrow's batch: streaming feels genuinely different from everything Airflow scheduled in step 9, expect to revisit topics and partitions before it settles.
You can run a distributed Spark job on a schedule, transform it with dbt, and watch data arrive in real time through Kafka instead of waiting for tomorrow's batch. Next up: pulling every layer into one platform you actually own.
Finish this section to unlock.
+100 XP
Take this if you're heading toward real-time replication work: Change Data Capture turns the database writes SQL taught you in step 3 into the same live event stream Kafka processes in step 11.
Unify what steps 7 through 12 built separately: the lakehouse pattern layers open table formats over cheap storage so batch, streaming, and warehouse workloads finally share one copy of data.
Reach for this if your stack runs on a cloud warehouse instead of open lakehouse tables: Snowflake takes the dimensional models from step 5 and the dbt transforms from step 10 and runs them at scale with zero infrastructure to manage.
Every pipeline from step 9's DAGs to step 13's lakehouse can keep running happily while serving stale or broken rows; freshness, volume, schema, and lineage checks are what tell you the data is wrong before a dashboard tells your CFO.
Pull every earlier step into one lifecycle view: the same modeling, moving, processing, and orchestrating skills practiced individually, now run as the repeatable platform a working data engineer owns end to end.
You can take raw source data from a dozen systems, model it, move it, process it at scale, and serve it through a lakehouse a whole company can trust, that's the Data Engineer badge, earned.
Finish this section to unlock.
+100 XP
Finish every required step, at least 70% of them genuinely done (not skipped), to earn this badge and 500 XP.