Backpropagation

Backpropagation, short for backward propagation of errors, is a core algorithm for training Artificial Neural Networks that efficiently computes the Gradient of a Loss Function with respect to model parameters, including Weights and Biases. It applies the Chain Rule from Calculus to recursively compute Partial Derivatives, propagating information from the output layer backward through the hidden layers. These gradients enable Optimization algorithms such as Stochastic Gradient Descent to adjust parameters to minimize the loss; backpropagation itself computes gradients rather than performing parameter updates. The technique was prominently popularized by the 1986 Nature paper “Learning representations by back-propagating errors,” coauthored by David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. It underpins the training of many Deep Learning architectures, including Convolutional Neural Networks and Transformers, with implementations optimized for hardware such as NVIDIA’s Graphics Processing Units and Google’s Tensor Processing Units.