Stop your fine-tune from imitating the prompt with loss masking

My training loss looks great, but the model just repeats one answer no matter what I ask it.

IntermediateRead 5 minDoing: about twenty minutes6 steps

▶ Start

What you'll have at the end

Training labels that show -100 on every prompt token, checked by printing one masked example before the real job runs

Before you start

You need

A supervised fine-tuning script that already trains without crashing, on a dataset shaped like a fixed instruction template plus a short answer (a label, a short reply, a classification word), and access to the exact tokenizer your base model uses.

Not covered

Picking a different dataset format or base model to raise a low-signal answer's word count, and the separate catastrophic-forgetting failure that can still show up after the prompt is masked correctly.

Leans on

6 steps

Where to go next

Catch an overfit fine-tune before you ship it

Once the mask is fixed and the loss reflects real learning, the natural next check is whether the model is memorizing the training rows instead of learning the task.

Checked 18 Aug 2026

Part of the LLM Fine-tuning cookbook