Gradient Descent Revisited — Slow Plateaus and Contour Maps

Start here Week 2 gave us gradient descent. This week asks: is it fast? The answer: not always. On flat regions of the error surface it crawls. Before we fix that with better optimisers, we learn a simple tool to see what an optimiser is doing — the contour map.

• Sigmoid neuron $f(x) = \frac{1}{1 + e^{-(wx + b)}}$, data (0.5, 0.2) and (2.5, 0.9).
• Loss $L(w, b) = (1/N) \sum (y_{i} - f(x_{i}))^{2}$. Guessing (w, b) by hand works for 2 parameters but not for millions.
• From the Taylor series: the loss falls fastest when we move opposite to the gradient.
• Update rule:$w_{t+1} = w_t - \eta \nabla w_t, b_{t+1} = b_t - \eta \nabla b_t.$
• For the sigmoid neuron: $\nabla w = \sum (f(x_{i}) - y_{i}) f(x_{i})(1 - f(x_{i})) x_{i}, \nabla b = \sum (f(x_{i}) - y_{i}) f(x_{i})(1 - f(x_{i}))$.

Full derivations are on the Week 2 page Gradient Descent.

The gradient is the slope Δy/Δx of the loss curve.

• Where the curve is steep, the gradient is large.
• Where the curve is gentle, the gradient is small.

Our update is $\eta \nabla w$. So:

• On steep parts → big steps.
• On gentle parts → tiny steps.

Lecture example — starting points Run gradient descent for $t_{\mathrm{max}} = 150$ steps from $(w, b) = (-2, 2)$, and again from (−3, 2). Whatever the starting point, once the path reaches a gentle region, progress slows down a lot. Starting from (−4, −4) is even worse — the loss there is almost flat.

Everyday example Rolling a marble on a nearly flat table: it barely moves, even though the floor (the minimum) is somewhere ahead.

3D plots are hard to read. A contour map is a 2D way to show the same surface.

How it’s made: take horizontal slices of the error surface at equal heights (equal loss values). Look from the top. Each slice becomes a closed line — a contour. Every point on one contour has the same loss.

How to read a contour map:

This is exactly like height lines on a trekking map of a hill.

Gradient descent on a contour map: the path always crosses contours at right angles (the gradient is perpendicular to the contour line). Where contours are far apart, the dots of the path bunch together — tiny steps.

Plain gradient descent:

• takes a lot of time to cross regions with a gentle slope,
• because the gradient there is very small.

Can we do better? Yes — momentum-based gradient descent (next page).

One more thing to keep in mind: every optimiser starts by computing $\nabla w_{0}$ at the starting point.

In one line: Gradient descent moves fast on steep slopes and crawls on gentle ones; contour maps show the error surface from the top, where close lines mean steep and far lines mean gentle.

Exam words: gentle slope, steep slope, plateau, contour, contour map, equal-loss line, top view.