New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
My training script died overnight and I don't know if resuming from the checkpoint will pick up cleanly or quietly start over.
What you'll have at the end
A completed fine-tune that reaches the original target step count, with the resumed loss curve lining up against the pre-crash one
You need
A Hugging Face Trainer fine-tuning script that was running unattended when it stopped, plus the output directory it was writing checkpoints to, with at least one checkpoint folder already saved before the crash.
Not covered
Choosing how often to checkpoint or where to store checkpoints before a run starts, and a crash that also corrupted the checkpoint files on disk rather than just interrupting a write.
Fix a CUDA out of memory crash with QLoRA
if the crash was a CUDA out of memory error rather than an unplanned process death, since that failure has its own real fix
Run a complete LoRA fine-tune on a free GPU
for a clean setup when nothing from the crash is actually usable and the run has to restart from scratch
Diagnose a fine-tuning loss curve that won't drop or keeps spiking
when the loss still looks wrong after a genuinely correct resume, since the cause is something else in training
Checked 18 Aug 2026
Part of the LLM Fine-tuning cookbook