Activation Function
A non-linear function applied to each neuron's output in a neural network, enabling the model to learn complex patterns beyond linear transformations.
An activation function is a non-linear transformation applied to a neuron's linear output z = Wx + b. Without it, stacking layers collapses into a single linear transformation, making non-linear problems unsolvable. The choice is one of the factors that shape training speed and final accuracy.
In computer vision models, ReLU is one representative choice for hidden layers. Defined as f(x) = max(0, x), it passes positive values unchanged and zeros out negatives. Sigmoid and tanh saturate where the input magnitude is large, whereas ReLU keeps a gradient of 1 across the positive range, and its forward pass reduces to a single comparison.
- ReLU:
f(x) = max(0, x). Simple to compute, with no gradient saturation on the positive side; a neuron receiving only negative inputs has a zero gradient and stops updating (dying ReLU problem) - Leaky ReLU:
f(x) = max(0.01x, x). Assigns a small slope to the negative range (the 0.01 in the formula is one coefficient choice and varies by implementation), so the gradient is not 0 for negative inputs - GELU: Smooth activation used in Transformers, approximated as x times the standard normal CDF. Adopted in BERT and Vision Transformer
- Softmax: Output layer function producing probability distributions across classes, applied to the output stage of multi-class image classification
For super-resolution and generation, output layers use tanh (range -1 to 1) or sigmoid (range 0 to 1) to constrain pixel values. The principle: ReLU variants for hidden layers, task-specific functions for outputs.