New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
My fine-tuning loss just sits flat, or spikes randomly, and I don't know if it's the learning rate, the data, or something else entirely.
What you'll have at the end
A loss curve back on a normal downward slope, with the exact fix written down: lower rate, longer warmup, or a clip threshold
You need
A fine-tuning run, in progress or already finished, whose trainer logs the loss for every step, and ideally the gradient norm too, somewhere you can pull the raw numbers from rather than only a smoothed chart.
Not covered
A model whose loss looks fine but whose answers are memorized or wrong, and picking your very first learning rate and warmup before you've ever pressed start: different problems with their own recipes.
Leans on
Pick starting values for LoRA rank, alpha, and learning rate
for the learning rate and warmup values to start from before your first run, rather than fixing a curve after the fact
Catch an overfit fine-tune before you ship it
once the curve is back on a normal slope, to check whether the model is really learning or quietly memorizing
Checked 18 Aug 2026
Part of the LLM Fine-tuning cookbook