Gradient Descent

Gradient Descent is a first-order iterative Optimization Algorithm used to minimize a differentiable Objective Function, typically a Loss Function, by updating the parameters of a Machine Learning model such as a Neural Network. It computes the Gradient, the vector of Partial Derivatives with respect to the parameters, which points in the direction of steepest local increase under the Euclidean Norm. Parameters are then updated in the negative-gradient direction, with the step scaled by a Hyperparameter called the Learning Rate. This update does not generally guarantee convergence to either a Local Minimum or a Global Minimum: behavior depends on the objective, initialization, learning-rate schedule, and other conditions. For Nonconvex Optimization, including most neural-network training, the method may approach a local minimum or Saddle Point, or fail to converge; standard global-minimum convergence guarantees require assumptions such as Convexity, appropriate regularity conditions, and suitable step sizes. Variants widely used in Deep Learning include Stochastic Gradient Descent, Mini-batch Gradient Descent, and adaptive methods such as Adam and RMSprop. Frameworks such as TensorFlow, developed by Google, and PyTorch, originally developed at Meta, use Automatic Differentiation, including Backpropagation, to compute these gradients efficiently through Computational Graphs.