Gradient Descent

Optimization · Axiom Academy

The fundamental first-order optimization algorithm for finding local minima 1. The Gradient Descent Update Rule Gradient descent iteratively updates the current point by moving in the direction opposite to the gradient. The update rule is beautifully simple: Here, x k is the current point, ∇f(x k ) is the gradient (direction of steepest ascent), and α is the step size (also called learning rate). The negative sign means we move against the gradient to descend. 2. Following the Steepest Descent Imagine standing on a hillside. The gradient points in the direction of steepest ascent - straight uphill. Gradient descent does the opposite: it takes steps in the direction of steepest descent , like water flowing downhill. Watch as gradient descent navigates the contour plot of a function, taking steps toward the minimum at the center: 3. The Critical Role of Step Size The step size α controls how far we move in each iteration. Too small, and convergence is painfully slow. Too large, and we might overshoot the minimum or even diverge! Watch how three different step sizes behave on the same problem: 4. Convergence and Local Minima Gradient descent converges to a point where the gradient is zero: ∇f(x*) = 0. This could be a local minimum, global minimum, or even a saddle point. For convex functions (bowl-shaped), any local minimum is also global. But for non-convex functions, gradient descent may get stuck in local minima: 5. When Gradient Descent Shines (and When It Struggles)

This is the written version of the interactive lesson above. See the full Optimization course.