PySpark Cheat Sheet

Updated 2026-04-20

Next Topic: Reverse ETL and Data Activation Cheat Sheet

🧠Study flashcards on this topic135 cards · spaced repetition→

PySpark is the Python API for Apache Spark, the distributed computing framework designed for large-scale data processing. With the release of Spark 4.0 — bringing ANSI SQL mode by default, the VARIANT data type, Python UDTFs, Spark Connect, and over 5,100 improvements — PySpark now offers an even richer set of tools for data scientists and engineers processing terabytes of data across clusters. Understanding the lazy evaluation model is critical: transformations build a logical plan, but nothing executes until an action is called, allowing Spark's Catalyst optimizer to generate the most efficient physical execution strategy. This approach makes PySpark both powerful for big data workloads and surprisingly accessible for those familiar with pandas-style operations.

What This Cheat Sheet Covers

This topic spans 31 focused tables and 314 indexed concepts, 135 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: SparkSession InitializationTable 2: Reading Data SourcesTable 3: DataFrame CreationTable 4: Schema DefinitionTable 5: Basic DataFrame OperationsTable 6: Column Selection and ManipulationTable 7: Filtering and Conditional LogicTable 8: AggregationsTable 9: Sorting and RankingTable 10: JoinsTable 11: Set OperationsTable 12: Null HandlingTable 13: String FunctionsTable 14: Math FunctionsTable 15: Date and Time FunctionsTable 16: JSON and Struct FunctionsTable 17: Array, Map, and Higher-Order FunctionsTable 18: Window FunctionsTable 19: Pivoting and ReshapingTable 20: User-Defined FunctionsTable 21: Partitioning and RepartitioningTable 22: Caching, Persistence, and CheckpointingTable 23: Writing DataTable 24: SQL IntegrationTable 25: RDD OperationsTable 26: Broadcast and AccumulatorsTable 27: Sampling and StatisticsTable 28: Machine Learning (MLlib)Table 29: Structured StreamingTable 30: Performance OptimizationTable 31: Configuration

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: SparkSession Initialization

Every PySpark program starts here — the SparkSession is the single entry point that wires your code to the cluster. The builder pattern below lets you name the app, point it at a cluster manager, tune resources like executor memory, and (since Spark Connect) even connect to a remote cluster from a lightweight client.

Method	Example	Description
SparkSession.builder	`spark = SparkSession.builder.appName("MyApp").getOrCreate()`	• Entry point for creating a Spark application • initializes or retrieves an existing SparkSession
config	`spark = SparkSession.builder.config("spark.executor.memory", "4g").getOrCreate()`	Sets Spark configuration properties like executor memory, parallelism, or shuffle partitions
master	`spark = SparkSession.builder.master("local[*]").getOrCreate()`	• Defines cluster manager URL • *local[]** uses all available cores on local machine
appName	`spark = SparkSession.builder.appName("DataPipeline").getOrCreate()`	Sets application name visible in Spark UI for monitoring and debugging
getOrCreate	`spark = SparkSession.builder.getOrCreate()`	Returns existing active session or creates new one if none exists
enableHiveSupport	`spark = SparkSession.builder.enableHiveSupport().getOrCreate()`	Enables Hive support for reading from and writing to Hive metastore tables
remote (Spark Connect)	`spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()`	• Connects to a remote Spark cluster via Spark Connect • lightweight client (~1.5 MB via `pip install pyspark-connect`)
active	`spark = SparkSession.active`	Returns the active SparkSession for the current thread without builder
newSession	`spark2 = spark.newSession()`	Creates new session with separate SQLConf and temp views but shared SparkContext
stop	`spark.stop()`	Terminates SparkSession and releases all cluster resources

Table 2: Reading Data Sources

Spark reads from almost anywhere, and the spark.read interface stays nearly identical across formats — only the method or the format() string changes. The practical takeaway is the format trade-off: CSV and JSON are convenient but slow to infer and parse, while columnar formats like Parquet and ORC are far faster for analytics, and Delta Lake layers ACID transactions and time travel on top.

Format	Example	Description
CSV	`df = spark.read.csv("data.csv", header=True, inferSchema=True)`	• Reads CSV files • inferSchema=True automatically detects column types but adds overhead

The rest of this section is part of the full cheat sheet

Table 3: DataFrame Creation

When your data isn't coming from a file, these methods build a DataFrame directly in memory — from Python lists, an existing RDD, a pandas DataFrame, or just a range of integers. They're the bread and butter of quick tests, prototypes, and the small lookup tables you assemble inline.

Method	Example	Description
createDataFrame	`df = spark.createDataFrame([(1, "a"), (2, "b")], ["id", "name"])`	Creates DataFrame from Python list, RDD, or pandas DataFrame

The rest of this section is part of the full cheat sheet

Table 4: Schema Definition

Letting Spark infer a schema is convenient but costs an extra pass over the data and can guess types wrong — so for production pipelines you usually define the schema yourself. These approaches range from the verbose, fully-explicit StructType to a concise DDL string, plus Spark 4.0's VariantType for storing semi-structured JSON-like data in an optimized binary form.

Approach	Example	Description
StructType	`from pyspark.sql.types import StructType, StructField, StringType` `schema = StructType([StructField("name", StringType(), True)])`	• Explicitly defines DataFrame schema • True indicates column is nullable

The rest of this section is part of the full cheat sheet

Table 1: SparkSession Initialization

Method	Example	Description
SparkSession.builder	`spark = SparkSession.builder.appName("MyApp").getOrCreate()`	• Entry point for creating a Spark application • initializes or retrieves an existing SparkSession
config	`spark = SparkSession.builder.config("spark.executor.memory", "4g").getOrCreate()`	Sets Spark configuration properties like executor memory, parallelism, or shuffle partitions
master	`spark = SparkSession.builder.master("local[*]").getOrCreate()`	• Defines cluster manager URL • *local[]** uses all available cores on local machine
appName	`spark = SparkSession.builder.appName("DataPipeline").getOrCreate()`	Sets application name visible in Spark UI for monitoring and debugging
getOrCreate	`spark = SparkSession.builder.getOrCreate()`	Returns existing active session or creates new one if none exists
enableHiveSupport	`spark = SparkSession.builder.enableHiveSupport().getOrCreate()`	Enables Hive support for reading from and writing to Hive metastore tables
remote (Spark Connect)	`spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()`	• Connects to a remote Spark cluster via Spark Connect • lightweight client (~1.5 MB via `pip install pyspark-connect`)
active	`spark = SparkSession.active`	Returns the active SparkSession for the current thread without builder
newSession	`spark2 = spark.newSession()`	Creates new session with separate SQLConf and temp views but shared SparkContext
stop	`spark.stop()`	Terminates SparkSession and releases all cluster resources

Table 2: Reading Data Sources

Format	Example	Description
CSV	`df = spark.read.csv("data.csv", header=True, inferSchema=True)`	• Reads CSV files • inferSchema=True automatically detects column types but adds overhead

Table 3: DataFrame Creation

Method	Example	Description
createDataFrame	`df = spark.createDataFrame([(1, "a"), (2, "b")], ["id", "name"])`	Creates DataFrame from Python list, RDD, or pandas DataFrame

Table 4: Schema Definition

Approach	Example	Description
StructType	`from pyspark.sql.types import StructType, StructField, StringType` `schema = StructType([StructField("name", StringType(), True)])`	• Explicitly defines DataFrame schema • True indicates column is nullable