Step Size Selection

Optimization · Axiom Academy

LESSON Step Size Selection in Optimization Understanding how learning rate affects convergence in gradient descent and iterative optimization methods 1. The Fundamental Update Rule In gradient descent, we iteratively update our position to minimize a function f(x). The update rule is: Here, α is the step size that scales the gradient. It controls how aggressively we move toward the optimum. The gradient ∇f(x k ) points in the direction of steepest ascent, so we move in the opposite direction to descend. 2. When Step Size Is Too Large If α is too large, we overshoot the minimum at each iteration. This leads to oscillations or even divergence, where the algorithm moves farther from the optimum instead of closer. In this scenario, the iterates bounce back and forth across the valley, potentially with increasing magnitude. The algorithm becomes unstable and fails to converge. 3. When Step Size Is Too Small Conversely, if α is too small, each iteration makes only tiny progress toward the minimum. While the algorithm will eventually converge (assuming the function is well-behaved), it requires many iterations and becomes computationally inefficient. The iterates creep slowly down the gradient, taking small, cautious steps. This is stable but wasteful of computational resources. 4. Fixed vs Adaptive Step Size Methods Fixed step size: Use the same α for all iterations. Simple but requires careful tuning and may be suboptimal.

This is the written version of the interactive lesson above. See the full Optimization course.