LoRA and Parameter-Efficient Fine-Tuning Cheat Sheet

Updated 2026-05-25

Next Topic: MCP Servers - Popular Concrete - Cheat Sheet

🧠Study flashcards on this topic93 cards · spaced repetition→

Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique that adapts large pretrained models by injecting trainable low-rank matrices into frozen model layers, drastically reducing memory and compute requirements. LoRA emerged in 2021 as practitioners sought ways to fine-tune billion-parameter models without the prohibitive costs of full fine-tuning — freezing the base model and training only 0.1–1% of parameters while achieving comparable or better performance. The key insight: fine-tuning updates often live in low-rank subspaces, meaning a full-rank weight update can be decomposed into two smaller matrices without sacrificing task adaptation quality. Today, LoRA and its PEFT family (QLoRA, DoRA, PiSSA, rsLoRA, GaLore, etc.) are standard practice for customizing LLMs, vision models, and multimodal systems — enabling practitioners to fine-tune 70B+ models on consumer GPUs and deploy hundreds of task-specific adapters in production. Understanding rank selection, alpha scaling, target modules, initialization strategies, and merging techniques is essential for maximizing performance while minimizing cost.

What This Cheat Sheet Covers

This topic spans 15 focused tables and 111 indexed concepts, 93 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Core LoRA ConceptsTable 2: LoRA Variants and ExtensionsTable 3: PEFT Methods ComparisonTable 4: Hyperparameter ConfigurationTable 5: Quantization TechniquesTable 6: Memory and Performance OptimizationTable 7: Training Best PracticesTable 8: Advanced LoRA TechniquesTable 9: Model Merging TechniquesTable 10: Deployment ConsiderationsTable 11: LoRA with Alignment and Advanced TrainingTable 12: Vision and Multimodal LoRATable 13: LoRA Variants ComparisonTable 14: Common Pitfalls and SolutionsTable 15: Tools and Libraries

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Core LoRA Concepts

The mathematical foundation of LoRA is surprisingly compact: two small matrices multiply to approximate the weight update, with initialization designed so the adapter contributes nothing at the start of training. Mastering these core concepts — rank, alpha, target modules, and the forward-pass formula — is the prerequisite for understanding every variant and extension.

Concept	Example	Description
LoRA (Low-Rank Adaptation)	$W' = W_0 + \Delta W = W_0 + BA$ where $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$	• Freezes pretrained weights $W_0$ and injects trainable low-rank matrices $A$ and $B$ into each layer • rank $r \ll \min(d,k)$ drastically reduces parameters.
Rank ($r$)	`r=8`, `r=16`, `r=64`	• Dimensionality of low-rank decomposition • controls adapter capacity — lower rank = fewer params but less expressiveness • typical range 4–256 depending on model size and task.
Alpha ($\alpha$)	`alpha=16` (if `r=8`)	• Scaling factor that controls update magnitude via $\frac{\alpha}{r}$ • commonly set to $\alpha = 2r$ as heuristic • higher alpha = stronger adaptation signal.
Weight Update Formula	$h = W_0 x + \frac{\alpha}{r} B A x$	• During forward pass, base model output $W_0 x$ is adjusted by scaled LoRA term $\frac{\alpha}{r} BA x$ • backward pass updates only $A$ and $B$.
Parameter Efficiency	7B model: 0.1–1% trainable params (few MB adapters vs 14GB base)	• LoRA reduces trainable parameters by 100–1000× compared to full fine-tuning • a 7B model with rank-8 LoRA trains ~2M params instead of 7B.
Target Modules	`target_modules=["q_proj", "v_proj", "k_proj", "o_proj"]`	• Specifies which linear layers receive LoRA • typically query/key/value/output projections in attention • can also target MLPs (`gate_proj`, `up_proj`, `down_proj`).
Adapter Merging	$W_{merged} = W_0 + BA$	• Combines trained LoRA weights into base model for inference • zero latency overhead — model behaves as if fully fine-tuned.
Matrix Initialization	$A \sim \mathcal{N}(0, \sigma^2)$, $B = 0$	• $A$ initialized with Kaiming uniform or Gaussian • $B$ starts at zero so initial $\Delta W = BA = 0$ — adapter contributes nothing at start.

Table 2: LoRA Variants and Extensions

The LoRA family has grown rapidly since 2021. Each variant addresses a specific limitation of vanilla LoRA — rank saturation, quantization error, weight decomposition quality, or parameter count — and most are drop-in replacements available through Hugging Face PEFT.

Method	Example	Description
QLoRA (Quantized LoRA)	`load_in_4bit=True` with NF4 quantization	• Combines LoRA with 4-bit NF4 quantization and paged optimizer states • enables fine-tuning 65B models on 24GB GPU • base model stored in 4-bit, gradients in higher precision.

The rest of this section is part of the full cheat sheet

Table 3: PEFT Methods Comparison

Choosing the right PEFT method is a trade-off between parameter count, quality ceiling, memory requirements, and inference overhead. This table gives the essential comparison across the most-used methods; the right choice depends on GPU memory, dataset size, and target accuracy.

Method	Example	Description
Full Fine-Tuning	Train all 7B params (14GB weights + 28GB optimizer states)	• Updates every parameter — highest accuracy ceiling but requires ~3× model size in memory (weights + gradients + optimizer) • prohibitive for 70B+ models; still gold standard when compute allows.

The rest of this section is part of the full cheat sheet

Table 4: Hyperparameter Configuration

LoRA's small number of hyperparameters belies their importance: rank and alpha together determine how strongly the adapter can reshape the model, while target modules determine which computations are touched. Starting with the defaults below and ablating systematically produces better results than guessing.

Parameter	Example	Description
Rank Selection Strategy	Start `r=16`, scale to `r=64` for complex tasks	• Lower rank (4–16) for simple tasks or small datasets • higher rank (64–256) for complex reasoning or large data • rule of thumb: start at 16, increase if underfitting.

The rest of this section is part of the full cheat sheet

Table 1: Core LoRA Concepts

Concept	Example	Description
LoRA (Low-Rank Adaptation)	$W' = W_0 + \Delta W = W_0 + BA$ where $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$	• Freezes pretrained weights $W_0$ and injects trainable low-rank matrices $A$ and $B$ into each layer • rank $r \ll \min(d,k)$ drastically reduces parameters.
Rank ($r$)	`r=8`, `r=16`, `r=64`	• Dimensionality of low-rank decomposition • controls adapter capacity — lower rank = fewer params but less expressiveness • typical range 4–256 depending on model size and task.
Alpha ($\alpha$)	`alpha=16` (if `r=8`)	• Scaling factor that controls update magnitude via $\frac{\alpha}{r}$ • commonly set to $\alpha = 2r$ as heuristic • higher alpha = stronger adaptation signal.
Weight Update Formula	$h = W_0 x + \frac{\alpha}{r} B A x$	• During forward pass, base model output $W_0 x$ is adjusted by scaled LoRA term $\frac{\alpha}{r} BA x$ • backward pass updates only $A$ and $B$.
Parameter Efficiency	7B model: 0.1–1% trainable params (few MB adapters vs 14GB base)	• LoRA reduces trainable parameters by 100–1000× compared to full fine-tuning • a 7B model with rank-8 LoRA trains ~2M params instead of 7B.
Target Modules	`target_modules=["q_proj", "v_proj", "k_proj", "o_proj"]`	• Specifies which linear layers receive LoRA • typically query/key/value/output projections in attention • can also target MLPs (`gate_proj`, `up_proj`, `down_proj`).
Adapter Merging	$W_{merged} = W_0 + BA$	• Combines trained LoRA weights into base model for inference • zero latency overhead — model behaves as if fully fine-tuned.
Matrix Initialization	$A \sim \mathcal{N}(0, \sigma^2)$, $B = 0$	• $A$ initialized with Kaiming uniform or Gaussian • $B$ starts at zero so initial $\Delta W = BA = 0$ — adapter contributes nothing at start.

Table 2: LoRA Variants and Extensions

Method	Example	Description
QLoRA (Quantized LoRA)	`load_in_4bit=True` with NF4 quantization	• Combines LoRA with 4-bit NF4 quantization and paged optimizer states • enables fine-tuning 65B models on 24GB GPU • base model stored in 4-bit, gradients in higher precision.

Table 3: PEFT Methods Comparison

Method	Example	Description
Full Fine-Tuning	Train all 7B params (14GB weights + 28GB optimizer states)	• Updates every parameter — highest accuracy ceiling but requires ~3× model size in memory (weights + gradients + optimizer) • prohibitive for 70B+ models; still gold standard when compute allows.

Table 4: Hyperparameter Configuration

Parameter	Example	Description
Rank Selection Strategy	Start `r=16`, scale to `r=64` for complex tasks	• Lower rank (4–16) for simple tasks or small datasets • higher rank (64–256) for complex reasoning or large data • rule of thumb: start at 16, increase if underfitting.