Overview: Activation Function

Fern My week has apparently become people nudging us about activation functions. Fair. We keep saying the phrase like everyone was born holding a tiny calculus textbook, so episode eight thirty-one is us finally slowing down.

Lintel Good. Because activation function is one of those tiny-looking ideas that secretly holds the whole stack together. Think of a flat rubber sheet with dots drawn on it. If all you can do is stretch, rotate, and slide that sheet, straight lines stay straight.

Fern Right.

Lintel But if you can add a crease, a fold, a kink, suddenly you can separate dots that were tangled together. That kink is the activation function. It gives the network a way to bend.

Fern Okay, that already helps. Before we go deeper, quick grounding. We did neural networks in episode six thirty-three, but a neural network is basically layers of little math units passing numbers forward. And deep learning, episode six ninety-two, is when you stack a lot of those layers so the internal representations get richer.

Lintel Exactly. Each little unit, or neuron, takes input numbers, multiplies them by learned weights, adds a bias, and emits a number. A layer is a bunch of those units doing that in parallel.

Fern Wait, so if I have a hundred layers doing that, why isn't that already powerful? A hundred layers sounds… you know, annoyingly expensive enough to deserve powers.

Lintel Because the core operation there is linear algebra. A matrix multiplication, or matmul, takes a vector, which is just a list of numbers, and pushes it through a matrix, which is a grid of numbers. If you chain only linear operations, the chain can be mathematically collapsed into one bigger linear operation.

Fern Wait, really?

Lintel Really. It looks deep on the diagram, but without something nonlinear in between, it behaves like one big straight-line transformation. You paid for the stack, but you got the personality of a ruler.

Fern Stop it.

Lintel No, this is why the kink matters. The activation function sits after the weighted sum. Instead of passing that number straight to the next layer, the network runs it through a nonlinear function first.

Fern So in our rubber-sheet picture, the matmul stretches the sheet, and the activation gives it a fold.

Lintel Yes. And once you repeat stretch, fold, stretch, fold, the model can carve much more complicated boundaries through data. That is the piece that lets it recognize patterns that are not cleanly separable by one straight cut.

Fern Okay okay.

Lintel The formal definition is boring in a useful way. An activation function is a mostly differentiable nonlinear function inserted between linear layers. Mostly differentiable means training can usually calculate a slope for it, which matters when the model learns.

Fern That word, differentiable, is where people start sliding off the chair. Give me the non-math version.

Lintel Training needs to know which direction to nudge the weights. We talked about gradient descent in episode six seventy-seven: the gradient says which way reduces error, and roughly how steep that improvement is. Backpropagation sends that blame signal backward through the layers.

Fern Mm-hm.

Lintel So an activation cannot be a totally random cliff with no usable slope. It needs enough smoothness, or enough piecewise smoothness, that the training process can still move weights in a sensible direction.

Fern And the classic one everyone names first is ReLU, right? Rectified Linear Unit. Which sounds like a gym machine that judges your posture.

Lintel It does. ReLU is almost comically simple: if the input is negative, output zero. If the input is positive, output the input itself. That little rule is fast, cheap, and good enough that it became a default in many hidden layers.

Fern So negative signals get shut off, positive signals keep going. That feels too crude. Why did that work so well?

Lintel Partly because it breaks linearity without making computation expensive. Partly because, on the positive side, the slope stays strong. Older functions like sigmoid can squash values so hard that the training signal gets tiny as it moves backward.

Fern Tiny how?

Lintel Yes. The learning signal fades as it travels backward, so earlier layers barely update. ReLU helped because positive values do not get squeezed into a narrow range. The signal has an easier path through deep networks.

Fern I like that. The fold is sharp enough to bend the sheet, but not so mushy that training loses the signal.

Lintel Careful, but yes. ReLU is not magic. It can have dead neurons. If a neuron keeps landing on the negative side, ReLU keeps outputting zero, and that unit may stop contributing.

Fern Oh no.

Lintel That is why you see variants. Leaky ReLU lets a small negative value through instead of hard zero. GELU, SiLU, Swish, and SwiGLU are smoother or gated versions used in modern networks, especially around transformer feed-forward layers.

Fern Swish is such a fake-sounding name.

Lintel It is real, and yes, it was named because the curve resembled a shoe logo. Activation-function naming is where serious math briefly becomes a sneaker shelf.

Fern Okay, that's good.

Lintel Back to the mechanism. Hidden-layer activations are mostly there to create those bends and preserve trainability. Output-layer activations do a different job, because the final layer has to express the type of answer you want.

Fern So if the model is answering yes-or-no, you might use sigmoid at the end because zero to one reads like probability-ish.

Lintel Right. For a multi-class choice, softmax turns raw scores into probabilities that sum to one. There is the softmax raccoon again, rummaging in the cabinets, but this time it belongs here.

Fern There it is.

Lintel And for regression, where the output is just a number, you may use a linear output, meaning no squashing at the end. Predicting a price, a temperature, or a continuous value does not always want a zero-to-one range.

Fern This is where the product brain lights up, because the activation choice is not just math taste. It changes what the system can express, how it trains, and what kind of answer comes out.

Lintel Yes, but I would keep the hype contained. In large models, activation functions are usually tiny in parameter size. The feed-forward networks and attention layers carry the big weight counts. The activation is small, but load-bearing.

Fern Classic us.

Lintel Look at named systems. AlexNet in twenty twelve helped make deep image networks feel real, and ReLU-style activations were part of that practical turn. Transformer models after twenty seventeen still use nonlinear activations inside their blocks, even though attention gets the headlines.

Fern And the examples are not only chat models. Whisper has an encoder-decoder setup for audio transcription. BERT-family embedding models are older encoder-style text systems. YOLO-style vision systems use modern activation choices too.

Lintel Exactly. The architecture changes by modality, but the pattern persists. You still have layers transforming vectors, and you still need nonlinear functions between those transformations so the stack does not collapse into one linear move.

Fern Okay, but where does this stand now? Is activation function an old lesson, or is it still something people actively care about?

Lintel Still core. No fake obituary. Modern networks absolutely still rely on activations. The research frontier may argue over which variant optimizes better, but the basic role has not been absorbed away by attention, routing, or bigger context windows.

Fern That feels like the honest version. The famous part of a transformer is attention, because it explains relationships between tokens. But the boring fold inside the feed-forward block is still doing work.

Lintel Yes. If one thing sticks, make it this: depth only becomes useful when the network can bend between layers. The activation function is the bend.

Fern Fine, Lintel. We have honored the tiny kink. Exploring Next survives another foundational math raccoon.