First-Order Necessary Conditions

Optimization · Axiom Academy

LESSON First-Order Necessary Conditions Understanding why the gradient must vanish at local optima 1. One-Dimensional Case: The Derivative Test In one dimension, a function f(x) has a local minimum or maximum at x* only if the derivative equals zero: f'(x*) = 0. This is because the derivative represents the rate of change - if it were positive or negative, we could decrease the function value by moving in the opposite direction. 2. The Gradient Points Downhill In multiple dimensions, the gradient vector points in the direction of steepest ascent. At any point x, moving in the direction of -∇f(x) (negative gradient) decreases the function value most rapidly. This is the fundamental principle behind gradient descent optimization. 3. Zero Gradient at Local Optima At a local minimum, the function cannot decrease in any direction. If the gradient were non-zero, we could move in the direction of -∇f(x) to find a lower value, contradicting the assumption that we're at a minimum. Therefore, at any local optimum (minimum or maximum), the gradient must vanish. Consider f(x, y) = x² + y². At the origin (0, 0), we have a minimum. The gradient is ∇f = (2x, 2y), which equals (0, 0) at the origin. Away from the origin, gradient vectors point radially outward, showing how the function increases in all directions from the minimum.

This is the written version of the interactive lesson above. See the full Optimization course.