LightGBM Cheat Sheet

Updated 2026-05-21

Next Topic: Loss Functions in Deep Learning Cheat Sheet

🧠Study flashcards on this topic89 cards · spaced repetition→

LightGBM (Light Gradient Boosting Machine) is a gradient boosting framework developed by Microsoft, introduced at NeurIPS 2017, and built for high-speed, memory-efficient training on large tabular datasets. It addresses the core bottleneck of traditional GBDT — the expensive exact-split search — through histogram-based learning, leaf-wise tree growth, Gradient-based One-Side Sampling (GOSS), and Exclusive Feature Bundling (EFB). The result is training speeds up to 20× faster than XGBoost with comparable accuracy. The critical mental model: unlike XGBoost's depth-wise growth, LightGBM grows the single leaf with the maximum loss reduction at each step — which converges faster but requires careful tuning of num_leaves and min_data_in_leaf to prevent extreme one-sided trees that overfit small datasets.

What This Cheat Sheet Covers

This topic spans 16 focused tables and 95 indexed concepts, 89 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Core Architecture — Histogram-Based Learning and Leaf-Wise GrowthTable 2: Boosting TypesTable 3: Key Parameters — Tree Structure and ComplexityTable 4: Key Parameters — Learning Rate, Iterations, and SamplingTable 5: Regularization ParametersTable 6: Objective Functions and MetricsTable 7: Python API — Native InterfaceTable 8: Python API — Scikit-learn InterfaceTable 9: Early Stopping and CallbacksTable 10: Feature Importance and SHAP InterpretabilityTable 11: Categorical Feature HandlingTable 12: GPU AccelerationTable 13: Handling Imbalanced DataTable 14: Distributed and Parallel LearningTable 15: Custom Objectives and MetricsTable 16: Advanced Parameters and Specialized Features

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Core Architecture — Histogram-Based Learning and Leaf-Wise Growth

LightGBM's speed advantage comes from two structural innovations that reshape how trees are built. Understanding these mechanisms makes every subsequent parameter decision more intuitive.

Technique	Example	Description
Histogram-based split finding	`max_bin=255 # default bin count`	Continuous features are discretized into integer bins (default 255), reducing split search from O(#data) to O(#bins) — the primary source of LightGBM's speed advantage.
Leaf-wise (best-first) tree growth	`# grows the single leaf with max delta loss`	• Expands the leaf that reduces loss the most at each step, rather than all leaves at a given depth • converges faster but can overfit without proper `num_leaves` and `min_data_in_leaf` guards
GOSS (Gradient-based One-Side Sampling)	`boosting='gbdt' # GOSS is default in gbdt`	• Keeps all large-gradient instances (more informative) and randomly samples small-gradient ones, rescaling sampled weights • maintains accuracy while training on a fraction of data
EFB (Exclusive Feature Bundling)	`# automatic; no param to set`	• Bundles mutually exclusive sparse features (rarely non-zero simultaneously) into single features using a greedy graph-coloring algorithm • reduces effective feature count near-losslessly
Histogram subtraction	`# sibling node hist = parent - current leaf`	Computes sibling node histograms by subtracting child histogram from parent, halving the work for building each tree split — only one of two siblings needs a full histogram pass.
max_bin	`params = {'max_bin': 255} # default` `# GPU: max_bin=63 recommended`	• Controls histogram resolution • higher = more accurate splits but slower training • reduce to 63–127 for GPU training or very large datasets without significant accuracy loss

Table 2: Boosting Types

The boosting parameter selects the ensemble construction algorithm. The choice fundamentally changes how trees are added and how overfitting is controlled — most practitioners start with gbdt and experiment from there.

Type	Example	Description
gbdt (Gradient Boosting Decision Tree)	`params = {'boosting': 'gbdt'}`	• Default • traditional gradient boosting — each tree corrects the residuals of the previous ensemble • Best general-purpose starting point for most tasks

The rest of this section is part of the full cheat sheet

Table 3: Key Parameters — Tree Structure and Complexity

These parameters directly control tree shape and are the most important knobs for balancing model expressiveness against overfitting in LightGBM's leaf-wise regime.

Parameter	Example	Description
num_leaves	`params = {'num_leaves': 31} # default` `# deep tree: 127, conservative: 15–31`	• Primary complexity control — maximum leaves per tree • always set smaller than `2^max_depth` • values 20–150 cover most use cases • too high causes overfitting

The rest of this section is part of the full cheat sheet

Table 4: Key Parameters — Learning Rate, Iterations, and Sampling

These parameters govern the boosting process speed, capacity, and stochasticity. Learning rate and num_leaves are typically the first two hyperparameters tuned in any LightGBM workflow.

Parameter	Example	Description
learning_rate	`params = {'learning_rate': 0.1} # default` `# fine-tuned: 0.01–0.05`	• Shrinkage applied to each tree's contribution • lower = more trees needed but often better generalization • aliases: `shrinkage_rate`, `eta`.

The rest of this section is part of the full cheat sheet

Table 1: Core Architecture — Histogram-Based Learning and Leaf-Wise Growth

LightGBM's speed advantage comes from two structural innovations that reshape how trees are built. Understanding these mechanisms makes every subsequent parameter decision more intuitive.

Technique	Example	Description
Histogram-based split finding	`max_bin=255 # default bin count`	Continuous features are discretized into integer bins (default 255), reducing split search from O(#data) to O(#bins) — the primary source of LightGBM's speed advantage.
Leaf-wise (best-first) tree growth	`# grows the single leaf with max delta loss`	• Expands the leaf that reduces loss the most at each step, rather than all leaves at a given depth • converges faster but can overfit without proper `num_leaves` and `min_data_in_leaf` guards
GOSS (Gradient-based One-Side Sampling)	`boosting='gbdt' # GOSS is default in gbdt`	• Keeps all large-gradient instances (more informative) and randomly samples small-gradient ones, rescaling sampled weights • maintains accuracy while training on a fraction of data
EFB (Exclusive Feature Bundling)	`# automatic; no param to set`	• Bundles mutually exclusive sparse features (rarely non-zero simultaneously) into single features using a greedy graph-coloring algorithm • reduces effective feature count near-losslessly
Histogram subtraction	`# sibling node hist = parent - current leaf`	Computes sibling node histograms by subtracting child histogram from parent, halving the work for building each tree split — only one of two siblings needs a full histogram pass.
max_bin	`params = {'max_bin': 255} # default` `# GPU: max_bin=63 recommended`	• Controls histogram resolution • higher = more accurate splits but slower training • reduce to 63–127 for GPU training or very large datasets without significant accuracy loss

Table 2: Boosting Types

Type	Example	Description
gbdt (Gradient Boosting Decision Tree)	`params = {'boosting': 'gbdt'}`	• Default • traditional gradient boosting — each tree corrects the residuals of the previous ensemble • Best general-purpose starting point for most tasks

Table 3: Key Parameters — Tree Structure and Complexity

These parameters directly control tree shape and are the most important knobs for balancing model expressiveness against overfitting in LightGBM's leaf-wise regime.

Parameter	Example	Description
num_leaves	`params = {'num_leaves': 31} # default` `# deep tree: 127, conservative: 15–31`	• Primary complexity control — maximum leaves per tree • always set smaller than `2^max_depth` • values 20–150 cover most use cases • too high causes overfitting

Table 4: Key Parameters — Learning Rate, Iterations, and Sampling

These parameters govern the boosting process speed, capacity, and stochasticity. Learning rate and num_leaves are typically the first two hyperparameters tuned in any LightGBM workflow.

Parameter	Example	Description
learning_rate	`params = {'learning_rate': 0.1} # default` `# fine-tuned: 0.01–0.05`	• Shrinkage applied to each tree's contribution • lower = more trees needed but often better generalization • aliases: `shrinkage_rate`, `eta`.