New: Intelligent Flashcards spaced-repetition decks for every topic, built to make things actually stick.
A standard feed-forward block applies one activation to a single projection. How is SwiGLU structured differently?
It applies Swish and then ReLU one after the other on a single projection
It averages the outputs of several independently trained sub-networks, like an ensemble
It runs two separate linear projections, sends one through Swish, then multiplies the two element-wise