Optical Flow
A technique that represents the apparent motion of each pixel between consecutive frames as a vector field. A foundational technology for video analysis, motion detection, and video stabilization.
Optical flow estimates the 2D motion vector of each pixel between two or more temporally consecutive images, capturing how objects and the camera move within a scene. It is a fundamental building block of video analysis, providing per-pixel or per-feature motion information that downstream algorithms use for tracking, segmentation, and prediction.
Optical flow computation rests on the brightness constancy assumption: a pixel's intensity does not change between frames over short time intervals. This assumption yields the optical flow constraint equation Ix*u + Iy*v + It = 0, where Ix, Iy are spatial gradients, It is the temporal gradient, and (u, v) is the flow vector to be estimated.
- Lucas-Kanade method: Assumes uniform flow within a local window and solves via least squares. Produces sparse flow at tracked feature points. Implemented in OpenCV as
cv2.calcOpticalFlowPyrLK(), it is the standard for real-time point tracking - Farneback method: Approximates each pixel's neighborhood with a quadratic polynomial to estimate dense flow fields.
cv2.calcOpticalFlowFarneback()returns a flow vector for every pixel in the frame - Deep learning approaches: CNN and Transformer-based methods like FlowNet, PWC-Net, and RAFT significantly outperform classical methods in accuracy. RAFT uses iterative refinement with a correlation volume to achieve state-of-the-art results on benchmarks like Sintel and KITTI
Applications span motion detection (combined with background subtraction), video stabilization (compensating camera shake), frame interpolation (generating intermediate frames for slow motion), action recognition, and autonomous driving (estimating surrounding motion). Real-time deployment leverages GPU implementations and lightweight architectures optimized for inference speed.