Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is an iterative Optimization Algorithm that minimizes a Loss Function by updating Model Parameters in the direction of a negative estimated Gradient, with step size controlled by a Learning Rate. Unlike Batch Gradient Descent, which computes each gradient using the entire Training Dataset, SGD estimates it from a single randomly sampled training example, or a small Mini-Batch in its widely used mini-batch formulation. This reduces computational cost per update, making SGD practical for large-scale Machine Learning and central to training Neural Networks in Deep Learning. The resulting Gradient Noise can help updates move away from Saddle Points and escape some Local Minima, although it does not guarantee finding a Global Minimum and can complicate Convergence. Extensions using Momentum and related adaptive optimization methods such as RMSprop and Adam can improve convergence speed and stability, depending on the problem and their configuration.