Newton's Method

Optimization · Axiom Academy

LESSON Newton's Method for Optimization Understanding second-order optimization through quadratic approximation and the Hessian matrix Newton's method starts by approximating the objective function f(x) with a second-order Taylor expansion around the current point x k : where ∇f(x k ) is the gradient (first derivatives) and H(x k ) is the Hessian matrix (second derivatives). This quadratic model captures both the slope and curvature of the function. To find the minimum of the quadratic approximation, we take its gradient and set it to zero: Solving for the optimal step Δx = x - x k gives us: This leads to the Newton's method update rule: The Hessian matrix H contains all second-order partial derivatives: The Hessian captures the curvature of the function. Its inverse H -1 provides a natural scaling of the gradient direction: Near the optimum, Newton's method exhibits quadratic convergence , meaning the error decreases quadratically at each iteration: where e k = ||x k - x*|| is the error at iteration k. This is much faster than the linear convergence of gradient descent! Newton's method is powerful but comes with important trade-offs: Quadratic convergence: Extremely fast near the solution Scale-invariant: Adapts to function curvature automatically Fewer iterations: Often reaches optimum in far fewer steps than first-order methods Hessian computation: O(n²) space and O(n³) time for inversion May not converge: Can diverge if Hessian is not positive definite

This is the written version of the interactive lesson above. See the full Optimization course.