New: Intelligent Flashcards spaced-repetition decks for every topic, built to make things actually stick.
Both BF16 and FP16 store numbers in 16 bits. What makes BF16 the safer default for training large models?
It runs only on older GPUs like the V100, where FP16 is not available
It devotes more bits to the mantissa than FP16, so it can represent each individual value with higher numeric precision
It keeps FP32's wide exponent range, so values rarely overflow or underflow and loss scaling is usually unnecessary