Gradient Descent

Calculus 3 · Axiom Academy

Follow the negative gradient downhill, one step at a time, to reach a function's minimum — the algorithm behind modern machine learning. For a function , the gradient points in the direction of steepest ascent — straight uphill. To go down , we step the other way, along . Take a small step, recompute the gradient, and step again; each move lands on a lower contour, closing in on the minimum. The gradient stacks the partial derivatives 2. Choosing the Learning Rate α The learning rate sets how far each step travels — and it is the whole game. Too timid and you crawl toward the minimum; too bold and you leap clean over the valley floor and climb the far wall, diverging instead of converging. Each step barely moves. You inch toward the minimum and may need thousands of iterations — safe, but slow. Steps overshoot the bottom and bounce onto higher ground on the far side. The value grows instead of shrinking — the iteration diverges. Large enough to make real progress, small enough to settle. The efficient "Goldilocks" zone — often around 0.01 to 0.1 . Methods like Adam and RMSProp tune automatically as the descent proceeds, for faster, steadier convergence. For f(x)=x^2 , one step multiplies x by . The iteration converges only when ; push past that and every step lands farther out than the last. Gradient descent never quite lands on the minimum — it approaches forever, in ever-smaller steps. So we pick a tolerance and stop once we are close enough.

This is the written version of the interactive lesson above. See the full Calculus 3 course.