XGBoost Cheat Sheet

Updated 2026-05-20

🧠Study flashcards on this topic93 cards · spaced repetition→

XGBoost (eXtreme Gradient Boosting) is a highly optimized, scalable implementation of gradient-boosted decision trees that consistently ranks among the top-performing algorithms in structured-data competitions and production ML systems. It solves regression, classification, ranking, and survival problems by sequentially fitting trees to residuals, with second-order Taylor expansion of the loss enabling both speed and strong regularization. The key mental model: XGBoost is not one algorithm — it is a framework; every major behavior from tree structure to sampling to the objective function is configurable, and nearly every real-world win comes from understanding which lever to pull first.

What This Cheat Sheet Covers

This topic spans 19 focused tables and 126 indexed concepts, 93 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: DMatrix — Data Loading and ConstructionTable 2: Core Training HyperparametersTable 3: Regularization ParametersTable 4: Objective FunctionsTable 5: Tree Method and Booster SelectionTable 6: Early Stopping and EvaluationTable 7: GPU TrainingTable 8: Feature Importance and SHAPTable 9: Handling Imbalanced DataTable 10: Native Categorical Feature SupportTable 11: Distributed Training — DaskTable 12: Distributed Training — PySparkTable 13: Hyperparameter Tuning with OptunaTable 14: Prediction Methods and Output TypesTable 15: Model Saving and LoadingTable 16: Monotone and Interaction ConstraintsTable 17: Multi-Output LearningTable 18: Learning to RankTable 19: Common Pitfalls and Best Practices

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: DMatrix — Data Loading and Construction

The DMatrix is XGBoost's native data container; all training, evaluation, and prediction flows through it. Feeding data via DMatrix rather than raw arrays enables efficient internal compression and avoids redundant work across boosting rounds.

Method	Example	Description
DMatrix from NumPy	`dtrain = xgb.DMatrix(X, label=y)`	Wraps a NumPy array or Pandas DataFrame with optional label, weight, and base_margin.
DMatrix from Pandas	`dtrain = xgb.DMatrix(df[feats], label=df['y'])`	• Accepts a `pd.DataFrame` • column names are preserved as feature names automatically
DMatrix from SciPy sparse	`dtrain = xgb.DMatrix(csr_matrix)`	• Accepts `scipy.sparse.csr_matrix` • implicit zeros are treated as missing, not as the value 0 — convert to dense if zeros are real values
DMatrix missing value	`dtrain = xgb.DMatrix(X, label=y, missing=np.nan)`	• Explicitly declares which value should be treated as missing • default is `np.nan`.
DMatrix with weights	`dtrain = xgb.DMatrix(X, label=y, weight=w)`	• Per-sample training weights • higher weights increase a sample's influence on gradient updates
DMatrix with base_margin	`dtrain = xgb.DMatrix(X, label=y,` `base_margin=prior_scores)`	• Per-sample initial prediction offset (raw margin, before link function) • overrides `base_score` when provided • used to warm-start from another model's output
DMatrix feature names	`dtrain = xgb.DMatrix(X, feature_names=['age','income'])`	Names are stored inside the DMatrix and propagate to feature importance plots automatically.
QuantileDMatrix	`dm = xgb.QuantileDMatrix(X, label=y)`	• Pre-quantizes data for the `hist` tree method, cutting memory use by ~8× (float64 → 1 byte per entry) • only compatible with `tree_method="hist"`.
DMatrix from CSV / LibSVM	`dtrain = xgb.DMatrix('train.csv?format=csv&label_column=0')`	• Loads files directly from disk • LibSVM format via `'train.libsvm'` • useful for large files that should not be loaded into Python memory
External memory (ExtMemQuantileDMatrix)	`dm = xgb.ExtMemQuantileDMatrix(it, max_bin=256)`	• Streams data in batches via a user-supplied iterator • batches are never fully loaded into RAM, enabling terabyte-scale training • requires `tree_method="hist"`.
set_info method	`dtrain.set_info(weight=w, feature_names=names)`	Updates metadata (label, weight, base_margin, group/qid, feature_names/types) on an already-created DMatrix without rebuilding it.

Table 2: Core Training Hyperparameters

These are the first parameters to understand and tune; they control tree complexity, learning speed, and the bias-variance tradeoff. Roughly 80% of XGBoost tuning effort lives in this table.

Parameter	Example	Description
n_estimators / num_boost_round	`n_estimators=500`	• Number of boosting rounds (trees). Use `n_estimators` in the sklearn API, `num_boost_round` in `xgb.train()` • functionally identical • Always set high and rely on early stopping

The rest of this section is part of the full cheat sheet

Table 3: Regularization Parameters

XGBoost's regularization operates at two levels: penalizing leaf weights directly (L1/L2) and penalizing tree complexity via the minimum gain threshold (gamma). Together they are the primary defense against overfitting without reducing model capacity.

Parameter	Example	Description
reg_lambda (lambda)	`reg_lambda=1.0`	• L2 regularization on leaf weights • default 1.0 • smooths weights without promoting sparsity • increase to shrink leaf values when overfitting

The rest of this section is part of the full cheat sheet

Table 4: Objective Functions

The objective defines what XGBoost optimizes and the link function applied to raw scores. Choosing the right objective is the single most consequential modeling decision.

Objective	Example	Description
reg:squarederror	`objective='reg:squarederror'`	• Regression with squared error (MSE) loss • the default for regression tasks

The rest of this section is part of the full cheat sheet

Table 1: DMatrix — Data Loading and Construction

Method	Example	Description
DMatrix from NumPy	`dtrain = xgb.DMatrix(X, label=y)`	Wraps a NumPy array or Pandas DataFrame with optional label, weight, and base_margin.
DMatrix from Pandas	`dtrain = xgb.DMatrix(df[feats], label=df['y'])`	• Accepts a `pd.DataFrame` • column names are preserved as feature names automatically
DMatrix from SciPy sparse	`dtrain = xgb.DMatrix(csr_matrix)`	• Accepts `scipy.sparse.csr_matrix` • implicit zeros are treated as missing, not as the value 0 — convert to dense if zeros are real values
DMatrix missing value	`dtrain = xgb.DMatrix(X, label=y, missing=np.nan)`	• Explicitly declares which value should be treated as missing • default is `np.nan`.
DMatrix with weights	`dtrain = xgb.DMatrix(X, label=y, weight=w)`	• Per-sample training weights • higher weights increase a sample's influence on gradient updates
DMatrix with base_margin	`dtrain = xgb.DMatrix(X, label=y,` `base_margin=prior_scores)`	• Per-sample initial prediction offset (raw margin, before link function) • overrides `base_score` when provided • used to warm-start from another model's output
DMatrix feature names	`dtrain = xgb.DMatrix(X, feature_names=['age','income'])`	Names are stored inside the DMatrix and propagate to feature importance plots automatically.
QuantileDMatrix	`dm = xgb.QuantileDMatrix(X, label=y)`	• Pre-quantizes data for the `hist` tree method, cutting memory use by ~8× (float64 → 1 byte per entry) • only compatible with `tree_method="hist"`.
DMatrix from CSV / LibSVM	`dtrain = xgb.DMatrix('train.csv?format=csv&label_column=0')`	• Loads files directly from disk • LibSVM format via `'train.libsvm'` • useful for large files that should not be loaded into Python memory
External memory (ExtMemQuantileDMatrix)	`dm = xgb.ExtMemQuantileDMatrix(it, max_bin=256)`	• Streams data in batches via a user-supplied iterator • batches are never fully loaded into RAM, enabling terabyte-scale training • requires `tree_method="hist"`.
set_info method	`dtrain.set_info(weight=w, feature_names=names)`	Updates metadata (label, weight, base_margin, group/qid, feature_names/types) on an already-created DMatrix without rebuilding it.

Table 2: Core Training Hyperparameters

These are the first parameters to understand and tune; they control tree complexity, learning speed, and the bias-variance tradeoff. Roughly 80% of XGBoost tuning effort lives in this table.

Parameter	Example	Description
n_estimators / num_boost_round	`n_estimators=500`	• Number of boosting rounds (trees). Use `n_estimators` in the sklearn API, `num_boost_round` in `xgb.train()` • functionally identical • Always set high and rely on early stopping

Table 3: Regularization Parameters

Parameter	Example	Description
reg_lambda (lambda)	`reg_lambda=1.0`	• L2 regularization on leaf weights • default 1.0 • smooths weights without promoting sparsity • increase to shrink leaf values when overfitting

Table 4: Objective Functions

The objective defines what XGBoost optimizes and the link function applied to raw scores. Choosing the right objective is the single most consequential modeling decision.

Objective	Example	Description
reg:squarederror	`objective='reg:squarederror'`	• Regression with squared error (MSE) loss • the default for regression tasks