New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
My fine-tuning script died with a CUDA out of memory error. Dropping the batch size to 1 didn't fix it.
What you'll have at the end
The same fine-tune finishing without a crash, using 4-bit QLoRA instead of 16-bit LoRA
You need
A 16-bit LoRA fine-tuning script for a roughly 38-billion-parameter model that's already crashing with a CUDA out of memory error on an 80 GB GPU, even after you dropped the per-device batch size down to 1.
Not covered
Picking the LoRA rank, alpha, or learning rate values themselves, and setting up a fine-tuning run from scratch on a GPU you haven't tried yet.
Pick starting values for LoRA rank, alpha, and learning rate
for real starting numbers on rank, alpha, and learning rate, if the adapter's own math turns out to be the oversized piece
Run a complete LoRA fine-tune on a free GPU
for a fresh run on a small GPU that starts at 4-bit from the first step instead of hitting this wall at all
Checked 15 Aug 2026
Part of the LLM Fine-tuning cookbook