Dropout
A regularization technique that randomly deactivates neurons during training, preventing co-adaptation and reducing overfitting by implicitly training an ensemble of sub-networks.
Dropout is a regularization technique that randomly sets a fraction of neuron activations to zero during each training step. A 2014 paper by Srivastava et al. reported that it substantially reduces overfitting by limiting co-adaptation between neurons. Dropped neurons are excluded from both forward and backward passes for that iteration.
Intuitively, dropout trains a different sub-network on each mini-batch, approximating an ensemble of exponentially many sub-networks. At inference, all neurons are active and outputs are scaled by (1 - drop rate). Alternatively, Inverted Dropout scales during training instead.
- Drop rate selection: Tuned to the task and the amount of data
- Spatial Dropout: Drops entire feature map channels rather than individual activations, respecting the spatial correlation structure of convolutional features
- DropPath (Stochastic Depth): Randomly skips entire residual blocks during training in ResNets and Vision Transformers
Extensions include DropKey (applied to attention weights) and R-Drop (enforcing output consistency between dropout masks).