Optimization, momentum and learning rates.
Learn through illustrated explanations, worked examples, practice questions and quick revision.
Before you start
Review the gradient-descent update and how gradients are computed in Weeks 2–3.
By the end: Compare batch, stochastic and mini-batch updates, then trace momentum and NAG.
Check the notation
η: learning rate; β: momentum coefficient; u: accumulated update. The notes show two conventions for placing η; keep one convention throughout a calculation.
Study sections
Follow the order or choose a sectionGradient Descent Revisited — Slow Plateaus and Contour Maps
Week 2 gave us gradient descent. This week asks: is it fast? The answer: not always. On flat regions of the error surface it crawls. Be…
02Momentum-Based Gradient Descent and Nesterov Accelerated Gradient
If someone keeps telling you “go left… go left… go left”, you start walking faster to the left. Momentum does exactly this: it remember…
03Stochastic and Mini-Batch Gradient Descent
With a million training examples, plain gradient descent does a million calculations just to take one step. That is very slow. Stochast…
04Choosing the Learning Rate and Momentum, and Line Search
The learning rate η decides how big every step is. Too small and training crawls; too big and it jumps around wildly. This page gives p…