Catastrophic forgetting wipes out skills your fine-tune never touched

After fine-tuning on support replies, my model started giving one-word answers to questions that had nothing to do with support.

IntermediateRead 5 minDoing: half a day6 steps

▶ Start

What you'll have at the end

A side-by-side probe-set score, base checkpoint against retrained checkpoint, proving the model's skills outside the support task work again, with the support task itself barely changed

Before you start

You need

A fine-tuned model already producing short, off-topic replies to unrelated questions, the exact pre-fine-tune base checkpoint still saved somewhere, and the training script and dataset that produced the fine-tune.

Not covered

Retraining on a different base model entirely, or algorithmic fixes like elastic weight consolidation that need custom training code beyond a standard fine-tuning script.

Leans on

6 steps

Where to go next

Catch an overfit fine-tune before you ship it

when the model nails the same support task suspiciously well instead of losing skills outside it

Pick starting values for LoRA rank, alpha, and learning rate

for adapter and learning-rate values least likely to cause this on the next run

Checked 15 Aug 2026

Part of the LLM Fine-tuning cookbook