Regression Analysis Cheat Sheet

Updated 2026-05-21

Next Topic: Set Theory and Mathematical Relations Cheat Sheet

🧠Study flashcards on this topic116 cards · spaced repetition→

Regression analysis is a set of statistical methods for estimating the relationships between a dependent variable and one or more independent variables, forming the backbone of predictive modelling and causal inference across every quantitative field. Its power lies not just in fitting a line, but in providing standard errors, hypothesis tests, and diagnostics that quantify how much trust to place in each estimate. The critical insight most practitioners learn too late is that a model's coefficients are only as meaningful as the assumptions it rests on — every regression analysis should be accompanied by careful diagnostics before results are reported.

What This Cheat Sheet Covers

This topic spans 18 focused tables and 146 indexed concepts, 116 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: OLS Foundations — The Simple Linear Regression ModelTable 2: OLS AssumptionsTable 3: Multiple Linear Regression — Matrix Form and Key FormulasTable 4: Regression DiagnosticsTable 5: Multicollinearity — Detection and RemediesTable 6: Model Selection CriteriaTable 7: Regularised Regression — Ridge, Lasso, and Elastic NetTable 8: Logistic RegressionTable 9: Multinomial and Ordinal Logistic RegressionTable 10: Poisson, Negative Binomial, and Count RegressionTable 11: Generalised Linear Models (GLM) — Link Functions and FamiliesTable 12: Robust RegressionTable 13: Time-Series Regression and AutocorrelationTable 14: Regression in PythonTable 15: Regression in RTable 16: Quantile RegressionTable 17: Advanced and Specialised Regression TopicsTable 18: Common Pitfalls and Reporting Standards

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: OLS Foundations — The Simple Linear Regression Model

The ordinary least squares estimator underpins almost every form of linear regression. Understanding how OLS works — what it minimises, what conditions make it optimal, and what its closed-form solution looks like — is the entry point to all more advanced regression methods.

Technique	Example	Description
Population regression model	$y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$	• The data-generating process relating the scalar response $y_i$ to predictor $x_i$ • $\varepsilon_i$ is the unobserved error term
OLS objective	$\min_{\beta_0,\beta_1} \sum_{i=1}^{n}(y_i - \beta_0 - \beta_1 x_i)^2$	• Minimises the sum of squared residuals • gives the line that is geometrically closest to all points in the vertical direction
OLS slope estimator	$\hat{\beta}_1 = \frac{\sum(x_i - \bar{x})(y_i - \bar{y})}{\sum(x_i - \bar{x})^2}$	• Closed-form solution obtained by setting the first-order conditions to zero • equals the sample covariance of $x$ and $y$ divided by the sample variance of $x$.
OLS intercept estimator	$\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}$	Ensures the fitted line always passes through the point of means $(\bar{x},\bar{y})$.
Fitted value and residual	$\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i$ $e_i = y_i - \hat{y}_i$	• $\hat{y}_i$ is the model's predicted value for observation $i$ • $e_i$ is the sample residual, not to be confused with the unobserved error $\varepsilon_i$
R-squared	$R^2 = 1 - \frac{SSR}{SST} = \frac{SSE}{SST}$	• Proportion of total variance in $y$ explained by the model • ranges from 0 to 1 but always increases with additional predictors regardless of relevance
Standard error of the regression (RSE)	$s = \sqrt{\frac{\sum e_i^2}{n-2}}$	• Estimate of the standard deviation of the error term • measures the average distance of observations from the fitted line
Gauss–Markov theorem	Under assumptions 1–5, $\hat{\boldsymbol{\beta}}$ is BLUE	When OLS assumptions hold, OLS is the Best Linear Unbiased Estimator — minimum variance among all linear unbiased estimators.

Table 2: OLS Assumptions

Every OLS result carries implicit guarantees only when specific conditions hold. Violating them does not always destroy the model, but it changes what remedies are needed and what claims can be made.

Assumption	Example	Description
Linearity in parameters	$y = \beta_0 + \beta_1 x^2 + \varepsilon$ is linear in $\beta$	• The model must be linear in the coefficients, not necessarily in the predictors • polynomial and log transforms are fine

The rest of this section is part of the full cheat sheet

Table 3: Multiple Linear Regression — Matrix Form and Key Formulas

With more than one predictor, matrix notation becomes necessary and reveals the structure of OLS elegantly. These formulas are the direct extensions of simple regression and underpin computation in every statistical package.

Technique	Example	Description
Matrix model	$\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}$	• $\mathbf{X}$ is the $n \times (p+1)$ design matrix with a column of ones for the intercept • $\boldsymbol{\beta}$ is a $(p+1) \times 1$ parameter vector

The rest of this section is part of the full cheat sheet

Table 4: Regression Diagnostics

Residual plots and influence statistics reveal whether the model's assumptions hold and whether individual observations are unduly distorting the fit. Running diagnostics is not optional — it is part of every defensible regression analysis.

Technique	Example	Description
Residuals vs Fitted plot	`plot(lm_fit, which=1)` in R	• Look for random scatter around zero • curvature signals nonlinearity • a fan pattern signals heteroscedasticity

The rest of this section is part of the full cheat sheet

Table 1: OLS Foundations — The Simple Linear Regression Model

Technique	Example	Description
Population regression model	$y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$	• The data-generating process relating the scalar response $y_i$ to predictor $x_i$ • $\varepsilon_i$ is the unobserved error term
OLS objective	$\min_{\beta_0,\beta_1} \sum_{i=1}^{n}(y_i - \beta_0 - \beta_1 x_i)^2$	• Minimises the sum of squared residuals • gives the line that is geometrically closest to all points in the vertical direction
OLS slope estimator	$\hat{\beta}_1 = \frac{\sum(x_i - \bar{x})(y_i - \bar{y})}{\sum(x_i - \bar{x})^2}$	• Closed-form solution obtained by setting the first-order conditions to zero • equals the sample covariance of $x$ and $y$ divided by the sample variance of $x$.
OLS intercept estimator	$\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}$	Ensures the fitted line always passes through the point of means $(\bar{x},\bar{y})$.
Fitted value and residual	$\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i$ $e_i = y_i - \hat{y}_i$	• $\hat{y}_i$ is the model's predicted value for observation $i$ • $e_i$ is the sample residual, not to be confused with the unobserved error $\varepsilon_i$
R-squared	$R^2 = 1 - \frac{SSR}{SST} = \frac{SSE}{SST}$	• Proportion of total variance in $y$ explained by the model • ranges from 0 to 1 but always increases with additional predictors regardless of relevance
Standard error of the regression (RSE)	$s = \sqrt{\frac{\sum e_i^2}{n-2}}$	• Estimate of the standard deviation of the error term • measures the average distance of observations from the fitted line
Gauss–Markov theorem	Under assumptions 1–5, $\hat{\boldsymbol{\beta}}$ is BLUE	When OLS assumptions hold, OLS is the Best Linear Unbiased Estimator — minimum variance among all linear unbiased estimators.

Table 2: OLS Assumptions

Every OLS result carries implicit guarantees only when specific conditions hold. Violating them does not always destroy the model, but it changes what remedies are needed and what claims can be made.

Assumption	Example	Description
Linearity in parameters	$y = \beta_0 + \beta_1 x^2 + \varepsilon$ is linear in $\beta$	• The model must be linear in the coefficients, not necessarily in the predictors • polynomial and log transforms are fine

Table 3: Multiple Linear Regression — Matrix Form and Key Formulas

Technique	Example	Description
Matrix model	$\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}$	• $\mathbf{X}$ is the $n \times (p+1)$ design matrix with a column of ones for the intercept • $\boldsymbol{\beta}$ is a $(p+1) \times 1$ parameter vector

Table 4: Regression Diagnostics

Technique	Example	Description
Residuals vs Fitted plot	`plot(lm_fit, which=1)` in R	• Look for random scatter around zero • curvature signals nonlinearity • a fan pattern signals heteroscedasticity